Multiplicity-free theorems of the Restrictions 
of Unitary Highest Weight Modules with 
respect to Reductive Symmetric Pairs 



Toshiyuki KOBAYASHI 

RIMS, Kyoto University, Kyoto 606-8502, Japan toshi@kurims.kyoto-u.ac.jp 

Summary. The complex analytic methods have found a wide range of applications 
in the study of multiplicity-free representations. This article discusses, in particular, 
its applications to the question of restricting highest weight modules with respect 
to reductive symmetric pairs. We present a number of multiplicity-free branching 
theorems that include the multiplicity-free property of some of known results such 
as the Clebsh-Gordan-Pieri formula for tensor products, the Plancherel theorem for 
Hermitian symmetric spaces (also for line bundle cases), the Hua-Kostant-Schmid 
K-type formula, and the canonical representations in the sense of Vershik-Gelfand- 
Graev. Our method works in a uniform manner for both finite and infinite dimen- 
sional cases, for both discrete and continuous spectra, and for both classical and 
exceptional cases. 

Key words: multiplicity-free representation, branching rule, symmetric pair, high- 
est weight module, Hermitian symmetric space, reproducing kernel, semisimple Lie 
group. 

Subject Classifications: Primary 22E46, Secondary 32A37, 05E15, 20G05, 53C35. 



Contents 

1 Introduction and statement of main results 2 

2 Main machinery from complex geometry 13 

3 Proof of Theorem A 18 

4 Proof of Theorem C 25 

5 Uniformly bounded multiplicities — Proof of Theorems B 
and D 28 

6 Counter examples 35 

7 Finite Dimensional Cases — Proof of Theorems E and F . . 41 

8 Generalization of the Hua Kostant Schmid Formula 47 



2 Toshiyuki KOBAYASHI 

9 Appendix: Associated Bundles on Hermitian Symmetric 

Spaces 61 

References 64 



1 Introduction and statement of main results 



The purpose of this article is to give a quite detailed account of the theory 
of multiplicity-free representations based on a non-standard method {visible 
actions on complex manifolds) through its application to branching prob- 
lems. More precisely, we address the question of restricting irreducible highest 
weight representations n of reductive Lie groups G with respect to symmetric 
pairs (G, H). Then, our main goal is to give a simple and sufficient condition 
on the triple (G, H, tt) such that the restriction 7r|jj is multiplicity-free. We 
shall see that our method works in a luiiform way for both infinite and finite 
dimensional representations, for both classical and exceptional cases, and for 
both continuous and discrete spectra. 

This article is an outgrowth of the manuscript [44] which I did not publish, 
but which has been circulated as a preprint. From then onwards, we have 
extended the theory, in particular, to the following three directions: 

1) the generalization of our main machinery (Theorem 2.2) to the vector 
bundle case ([49]), 

2) the theory of 'visible actions' on complex manifolds ([50, 51, 52]), 

3) 'multiplicity-free geometry' for coadjoint orbits ([53]). 

We refer the reader to our paper [47] for a precise statement of the general 
results and an exposition of the related topics that have recently developed. 

In this article, we confine ourselves to the line bundle case. On the one 
hand, this is sufficiently general to produce many interesting consequences, 
some of which are new and some others may be regarded as prototypes of 
various multiplicity- free branching theorems (e.g. [5, 10, 46, 54, 58, 66, 68, 81, 
90, 92]). On the other hand, the line bundle case is sufficiently simple, so that 
we can illustrate the essence of our main ideas without going into technical 
details. Thus, keeping the spirit of [44], we have included here the proof of our 
method (Theorem 2.2), its applications to multiplicity- free theorems (Theo- 
rems A-F), and the explicit formulae (Theorems 8.3, 8.4, and 8.11), except 
that we referred to another paper [50] for the proof of some algebraic lemmas 
on the triple of involutions of Lie algebras (Lemmas 3.6 and 7.5). 



1.1 Definition of multiplicity- free representations 

Let us begin by recalling the concept of the multiplicity-free decomposition of 
a unitary representation. 

Suppose H is a. Lie group of type I in the sense of von Neumann algebras. 
Any reductive Lie group is of type I as well as any algebraic group. We denote 
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by H the unitary dual of H, that is, the set of equivalence classes of irreducible 
unitary representations of H. The unitary dual H is endowed with the Fell 
topology. 

Suppose that (7r,H) is a unitary representation of H defined on a (sec- 
ond countable) Hilbcrt space H. By a theorem of Mautner, tt is decomposed 
uniquely into irreducible unitary representations of H in terms of the direct 
integral of Hilbert spaces: 

7r~ / m^{n)nda{iJ,) , (l-l-l) 
Jh 

where da{^) is a Borel measure on H, and the multiplicity function : H — > 
N U {00} is uniquely defined almost everywhere with respect to the measure 
da. 

Let End(W) be the ring of continuous operators on H, and Endiir(W) the 
subring of iJ-intertwining operators, that is, the commutant of {'K{g) : g G H} 
in End(7i:). 

Definition 1.1. We say that the unitary representation (tt, Ti) is multiplicity- 
free if the ring EndH(W) is commutative. 

It is not diflacult to see that this definition is equivalent to the following 
property: 

m-Kifj) < 1 for almost all e with respect to the measure dcr(/u) 

by Schur's lemma for unitary representations. In particular, it implies that 
any irreducible unitary representation oi H occurs at most once as a sub- 
representation of TT. 



1.2 Multiplicities for inductions and restrictions 

With regard to the question of finding irreducible decompositions of uni- 
tary representations, there are two fimdamental settings: one is the induced 
representation from smaller groups (e.g. harmonic analysis on homogeneous 
spaces), and the other is the restriction from larger groups (e.g. tensor product 

representations) . 

To be more rigorous, suppose G is a Lie group, and H is a closed sub- 
group of G. The G-irreducible decomposition of the induced representation 

L2-lndg T (t e H) is called the Plancherel formula^ while the _ff-irrcduciblc 
decomposition of the restriction 7r|i/ (tt e G) is referred to as the branching 
law. 

This subsection examines multiplicities in the irreducible decomposition 
of the induction and the restriction for reductive symmetric pairs (G, H) (see 
Subsection 3.1 for definition). 

Let us start with the induced representation. Van den Ban [2] proved that 
the multiplicity in the Plancherel formula for L^-lnd^ t is finite as far as 



4 Toshiyuki KOBAYASHI 



dimr < c». In particular, this is the case if r is the trivial representation 
1. Over the past several decades, the induced representation L^-Ind^ 1 has 
developed its own identity (harmonic analysis on reductive symmetric spaces 
G/H) as a rich and meaningful part of mathematics. 

In contrast, the multiplicities of the branching law of the restriction 'k\h 
{tt G G) are usually infinite. For instance, we saw in [36] that this is the case 
if (G, H) = {GL(j> + q, M), GL(p, M) x GL{q, M)) where mm{p, q) > 2, for any 
tempered representation tt of G. In this article, we illuminate by Example 6.3 
this wild behavior. 

In light of such a wild phenomenon of branching laws for reductive sym- 
metric pairs (G, H) with H non-compact, we proposed in [38, 40] to seek for a 
'nice' class of the triple (G, H, tt) in which a systematic study of the restriction 
7r|jj could be launched. 

Finiteness of multiplicities is a natural requirement for this program. By 
also imposing discrete decomposability on the restriction n\H, we established 
the general theory for admissible restriction in [38, 40, 41] and found that 
there exist fairly rich triples {G,H,7r) that enjoy this nice property. It is 
noteworthy that new interesting directions of research in the framework of 
admissible restrictions have been recently developed by M. Duflo, D. Gross, 
J.-S. Huang, J.-S. Li, S.-T. Lee, H.-Y. Loke, T. Oda, P. Pandzic, G. Savin, 
B. Speh, J. Vargas, D. Vogan, and N. Wallach (see [45, 48] and references 
therein) . 

Multiplicity-freeness is another ideal situation, in which we may expect an 
especially simple and detailed study of the branching law of tt\h- Thus, we 
aim for principles that lead us to abundant family of multiplicity- free cases. 

Among them, a well-known one is the dual pair correspondence, which has 
given fruitful examples in infinite dimensional theory in the following setting: 

a) G is the metaplectic group, and tt is the Weil representation. 

b) H = Hi ■ H2 forms a dual pair, that is, Hi is the commutant of H2 in 
G, and vice versa. 

This paper uses a new principle that generates multiplicity-free represen- 
tations. The general theory discussed in Section 2 brings us to uniformly 
bounded multiplicity theorems (Theorems B and D) and multiplicity-free the- 
orems (Theorems A, C, E and F) in the following setting: 

a) TT is a unitary highest weight representation of G (see Subsection 1.3), 

b) {G,H) is a symmetric pair (see Subsection 1.4). 

We note that we allow the case where continuous spectra occur in the 
branching law, and consequently, irreducible summands are not always highest 
weight representations. 

We remark that our bounded multiplicity theorems for the restriction 7r|jj 
(tt: highest weight module) may be regarded as the counterpart of the bounded 
multiplicity theorem for the induction L^-Ind^ r (r: finite dimensional repre- 
sentation) due to van den Ban. 
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1.3 Unitary highest weight modules 

Let us recall the basic notion of highest weight modules. 

Let G be a non-compact simple Lie group, a Cartan involution of G, and 
K := {g G G : 9g = g}. We write g = t + p for the Cartan decomposition of 
the Lie algebra g of G, corresponding to the Cartan involution 9. 

We assume that G is of Hermitian type, that is, the Riemannian sym- 
metric space G/K carries the structure of a Hermitian symmetric space, or 
equivalently, the center c(t) of t is non-trivial. The classification of simple Lie 
algebras g of Hermitian type is given as follows: 

su(p,9), ■Sp("-,R), so(m,2) (m ^ 2) , e6(_i4) , e7(_25) • 

Such a Lie algebras g satisfies the rank condition: 

rankG = rank/r, (1.3.1) 

or equivalently, a Cartan subalgebra of t becomes a Cartan subalgebra of 
0. By a theorem of Harish- Chandra, the rank condition (1.3.1) is equivalent 
to the existence of (relative) discrete series representations of G. Here, an 
irreducible unitary representation (tt, W) is called a (relative) discrete series 
representation of G if the matrix coefficient g {TT{g)u, v) is square integrable 
on G (modulo its center) for any u, w € W. 

If is a simple Lie algebra of Hermitian type, then there exists a charac- 
teristic element Z e c(6) such that 

fic :=fioc = ecep+ep- (1.3.2) 

is the cigcnspace decomposition of &d{Z) with eigenvalues 0, and —^/^, 
respectively. We note that dim c(6) = 1 if g is a simple Lie algebra of Hermitian 
type, and therefore c(fi) = RZ. 

Suppose V is an irreducible (flc, -f^)-inodule. We set 

:= {v gV :Yv = for any Y G p+} . (1.3.3) 

Since K normalizes p+, y+ is a if-submodule. Further, is either zero or 
an irreducible finite dimensional representation of K. We say ^ is a highest 
weight module if y + ^ {0}. 

Definition 1.3. Suppose tt is an irreducible unitary representation of G on a 
Hilbert space Ti. We set Hk ■= {v gH : dime C-span{7r(A:)i' : k € K} < oo}. 
Then, Hk is a dense subspace of H, on which the differential action dn of 
the Lie algebra g (and consequently that of its complexified Lie algebra gc) 
and the action of the compact subgroup K is well-defined. We say Hk is the 
underlying {gc,K)-module of (tt^H). We say (tt^H) is a unitary highest weight 
representation of G if =/= {0}. Then, tt is of scalar type (or of scalar min- 
imal K-type) if is one dimensional; tt is a (relative) holomorphic discrete 
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series representation for G if the matrix coefScient g i— > {TT{g)u, v) is square 
integrable on G modulo its center for any u,v G Ti.. Lowest weight mochiles 
and anti-holomorphic discrete series representations are defined similarly with 
p+ replaced by p_. 

This definition also applies to G which is not simple (sec Subsection 8.f). 

The classification of irreducible unitary highest weight representations was 
accomplished by Enright- Howc-Wallach [12] and H. Jakobscn [30] indepen- 
dently; see also [13]. There always exist infinitely many (relative) holomorphic 
discrete scries representations of scalar type for any non-compact simple Lie 
group of Hermitian type. 

1.4 Involutions on Hermitian symmetric spaces 

Suppose G is a non-compact simple Lie group of Hermitian type. Let t be an 
involutive automorphism of G commuting with the Cartan involution 6. We 

use the same letter r to denote its differential. Then r stabilizes B and also 
c(6). Because = id and c({) = RZ, we have the following two possibilities: 



Geometric meanings of these conditions become clear in the context of the 
embedding G^K^ ^ G/K, where G^ := {g G G : rg = g} and := G^nK 
(see [14, 27, 28, 35]). The condition (1.4.1) implies: 

1-a) T acts holomorphically on the Hermitian symmetric space G/K, 

1- b) G'^ /K'^ ^ G/K defines a complex submanifold, 
whereas the condition (1.4.2) implies: 

2- a) T acts anti-holomorphically on G/K, 

2-b) G'^ /K'^ ^ G/K defines a totally real submanifold. 

Definition 1.4. We say the involutive automorphism r is of holomorphic type 
if (1.4.1) is satisfied, and is of anti-holomorphic type if (1.4.2) is satisfied. The 
same terminology will be applied also to the symmetric pair (G, H) (or its Lie 
algebras (g, f))) corresponding to the involution r. 

Here, we recall that (G, H) is called a symmetric pair corresponding to r 
if H is an open subgroup of G'^ (see Subsections 3.1 and 3.2). We note that 
the Lie algebra f) of H is equal to := {X e : tX = X}. The classification 
of symmetric pairs (0,g^) for simple Lie algebras g was accomplished by M. 
Berger [6]. The classification of symmetric pairs (0,0^) of holomorphic type 
(respectively, of anti-holomorphic type) is regarded as a subset of Berger's 
list, and will be presented in Table 3.4.1 (respectively. Table 3.4.2). 



tZ = Z, 
tZ = -Z. 



(1.4.1) 
(1.4.2) 
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1.5 Multiplicity- free restrictions — infinite dimensional case 

We are ready to state our main results. Let G be a non-compact simple Lie 
group of Hermitian type, and (G, H) a symmetric pair. 

Theorem A (multiplicity- free restriction). Ifn is an irreducible unitary high- 
est weight representation of scalar type of G, then the restriction win is 
multiplicity-free. 

The branching law of the restriction ttIh may and may not contain discrete 
spectra in Theorem A. If (G, H) is of holomorphic type then the restriction 
7r|if is discretely decomposable (i.e. there is no continuous spectrum in the 
branching law); see Fact 5.1. Besides, the following theorem asserts that the 
multiplicities are still uniformly bounded even if we drop the assumption that 
TT is of scalar type. 

Theorem B (uniformly bounded multiplicities). We assume that the sym- 
metric pair (G, H) is of holomorphic type. Let n be an irreducible unitary 
highest weight representation ofG. 

1) The restriction '!t\h splits into a discrete Hilbert sum of irreducible unitary 
representations of H: 



2) C{-k) = 1 if IT is of scalar type. 

The second statement is a direct consequence of Theorems A and B (1). 
As we shall see in Section 6, such uniform boundedness theorem does not hold 
in general if tt is not a highest weight representation (see Examples 6.2 and 



Here are multiplicity-free theorems for the decomposition of tensor prod- 
ucts, which are parallel to Theorems A and B: 

Theorem C (multiplicity- free tensor product). Let tti and 7r2 be irreducible 
unitary highest (or lowest) weight representations of scalar type. Then the 
tensor product iti®'it2 is multiplicity-free as a representation of G. 

Here, 1:1(^712 stands for the tensor product representation of two unitary 
representations {'ni,'Hi) and {1^2, 'H2) realized on the completion Tii (8)7^2 of 
the pre-Hilbert space H\®'H2. (We do not need to take the completion if at 
least one of Wi or 7^2 is finite dimensional.) Theorem C asserts that multiplici- 
ties in the direct integral of the irreducible decomposition are not greater than 




and the multiplicities are uniformly bounded: 



C{tt) := sup m^{n) < 00 . 



6.3). 
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one in both discrete and continuous spectra. We note that continuous spectra 
appear in the irreducible decomposition of the tensor product representation 
7ri(8)7r2 only if 

TTi is a highest weight representation, and 
7r2 is a lowest weight representation, 

or in reverse order. 

If TTi and TT2 are simultaneously highest weight representations (or simul- 
taneously lowest weight representations), then the tensor product 7ri®7r2 de- 
composes discretely. Dropping the assumption of "scalar type" , we have still 
a uniform estimate of multiplicities: 

Theorem D (uniformly bounded multiplicities). Let tti and 1:2 be two irre- 
ducible unitary highest weight representations ofG. 

1) The tensor product 7ri(g)7r2 splits into a discrete Hilbert sum of irreducible 
unitary representations of G: 

TTI §772 ^ ^ 

and the multiplicities m^^^^^dj,) are uniformly bounded: 
C(7ri,7r2) := sup m^^^-^^iiJ.) < 00. 

2) C(7ri, 772) = 1 if both tti and ^2 are of scalar type. 

Remark 1.5. For classical groups, we can relate the constants C(7r) and 
C(7ri,7r2) to the stable constants of branching coefficients of finite dimen- 
sional representations in the sense of F. Sato [77] by using the see-saw dual 
pair correspondence due to R. Howe [23]. 

Our machinery that gives the above multiplicity-free theorems is built on 
complex geometry, and we shall explicate the general theory for the line bun- 
dle case in Section 2. The key idea is to transfer properties on representations 
(e.g. unitarity, multiplicity-freeness) into the corresponding properties of re- 
producing kernels, which we analyze by geometric methods. 

1.6 Multiplicity-free restrictions — finite dimensional case 

Our method yields multiplicity-free theorems not only for infinite dimensional 
representations but also for finite dimensional representations. 

This subsection presents multiplicity-free theorems that are regarded as 
'finite dimensional version' of Theorems A and C. Then' give a unified explana- 
tion of the multiplicity-free property of previously known branching formulae 
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obtained by combinatorial methods such as the Littlewood-Richardson rule, 
Koike- Tcrada's Young diagrammatic methods, Littclmann's path method, 
minor summation formulae, etc. (see [25, 55, 62, 68, 73, 80] and references 
therein). They also contain some 'new' cases, for which there are, to the best 
of our knowledge, no explicit branching formulae in the literature. 

To state the theorems, let Qc be a complex simple Lie algebra, and j a Car- 
tan subalgebra. We fix a positive root system Z\+(gc, j), and write ai, . . . , 
for the simple roots. Let tJi , . . . , cij„ be the corresponding fundamental weights. 
We denote by tta = tt^*^ the irreducible finite dimensional representation of Qc 
with highest weight A. 

We say Tr\ is of pan type if A is a scalar multiple of some LUi such that the 
nilradical of the maximal parabolic subalgebra corresponding to is abelian 
(see Lemma 7.3.1 for equivalent definitions). 

Theorem E (multiplicity- free restriction — finite dimensional case). Let tt 

be an arbitrary irreducible finite dimensional representation of Qc of pan type, 
and (flci f)c) be any symmetric pair. Then, the restriction 7r\f,^ is multiplicity- 
free. 

Theorem F (multiplicity- free tensor product — finite dimensional case) . The 
tensor product 17x^^2 of any two irreducible finite dimensional representations 
TTi and 772 of pan type is multiplicity-free. 

Theorems E and F are the counterpart to Theorems A and C for finite 
dimensional representations. The main machinery of the proof is again Theo- 
rem 2.2. 

Alternatively, one could verify Theorems E and F by a classical technique: 
finding an open orbit of a Borel subgroup. For example, Littelmann [61] and 
Panyushev independently classified the pair of maximal parabolic subalgebras 
(pijp2) such that the diagonal action of a Borel subgroup B of a complex 
simple Lie group Gc on Gc/Pi x G^jPi has an open orbit. Here, P\,Pi are 
the corresponding maximal parabolic subgroups of Gc- This gives another 
proof of Theorem F. 

The advantage of our method is that it enables us to understand (or even 
to discover) the multiplicity- free property simultaneously, for both infinite and 
finite dimensional representations, for both continuous and discrete spectra, 
and for both classical and exceptional cases by the single principle. This is 
because our main machinery (Theorem 2.2) uses only a local geometric as- 
sumption (see Remark 2.3.2 (2)). Thus, we can verify it at the same time for 
both compact and non-compact complex manifolds, and in turn get finite and 
infinite dimensional results, respectively. 

Once we tell a priori that a representation is multiplicity-free, we may be 
tempted to find explicitly its irreducible decomposition. Recently, S. Okada 
[68] found explicit branching laws for some classical cases that arise in The- 
orems E and F by using minor summation formulae, and H. Alikawa [1] for 
(fl) f)) = (^6, f4) corresponding to Theorem E. We note that the concept of pan 
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type representations includes rectangular- shaped representations of classical 
groups (sec [58, 68]). 

There are also some few cases where tti (8)772 is multiplicity- free even though 
neither tti nor 7r2 is of pan type. See the recent papers [46] or [81] for the 
complete list of such pairs (7ri,7r2) for gc = gl{n,C). The method in [46] to 
find all such pairs is geometric and based on the 'vector bundle version' of 
Theorem 2.2 proved in [49], whereas the method in [81] is combinatorial and 
based on case- by-case argument. 

We refer the reader to our papers [50, 51, 52] for some further results 
relevant to Theorems E and F along the same line of argument here. 

1.7 SL2 examples 

We illustrate the above theorems by SL2 examples. 

Exam,ple 1.7. 1) We denote by 7r„, the holomorphic discrete scries representa- 
tion of G = SL{2,M.) with minimal K-type Xn {n > 2), where we write Xn 
for the character of if = SO{2) parametrized by n e Z. Likewise 7r_„ denotes 
the anti-holomorphic discrete series representation of SL{2,M.) with minimal 
if -type x-n {n > 2). We note that any holomorphic discrete series of SL{2, R) 
is of scalar type. 

We write tt^— ^-^ (e = ±l,z/ G R) for the unitary principal series repre- 
sentations of S'L(2,M). We have a unitary equivalence f^^^ ~ ^1^^^- We 
write Xf for the unitary character of SOo{l, 1) ~ K parametrized by C € K. 

Let m > n > 2. Then, the following branching formulae hold. All of them 
are multiplicity-free, as is 'predicted' by Theorems A and C: 

(L7.1) (a) 
(L7.1) (b) 

e Yl '^m-n-2k , (1-7.1) (c) 

feeN 

0<2fe<m-n-2 

(1-7.1) (d) 

The key assumption of our main machinery (Theorem 2.2) that leads us to 
Theorems A and C is illustrated by the following geometric results in this SL2 
case: 

i) Given any clement z in the Poincare disk D, there exists G R such that 
g\/^y^ — j_ fact, one can take (p = —2a.vgz. This is the geometry that 
explains the multiplicity-free property of (1-7.1) (b). 

ii) Given any two elements z,w ^ D, there exists a linear fractional transform 
T on such that T{z) = z and T{w) = w. This is the geometry for (1.7.1) (d). 



TTnlsOoCl,!) 



/oo 
-00 



'^n\sO{2) — ^ Xn+2k , 

ken 

7r„§7r_„ ~ J TT^^ du 
^ ^ — \® 

T^m^T^n — ^ T^m+n+2k - 
feeN 
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These are examples of the geometric view point that we pursued in [50] 
for symmetric pairs. 

2) Here is a "finite dimensional version" of the above example. Let 7r„ be 
the irreducible n + 1-dimensional representation of SU{2). Then we have the 
following branching formulae: For m, n e N, 

7r„Uo(2) -^Xn® Xn-2 8 • • • %-„ , (1.7.1) (e) 

(1.7.1) (f) 

The formula (1-7.1) (e) corresponds to the character formula, whereas (1.7.1) (f) 
is known as the Clcbsch Gordan formula. The multiplicity-free property of 
these formulae is the simplest example of Theorems E and F. 

1.8 Analysis on multiplicity-free representations 

Multiplicity-free property arouses our interest in developing beautiful anal- 
ysis on such representations, as we discussed in Subsection 1.6 for finite di- 
mensional cases. This subsection picks up some recent topics about detailed 
analysis on multiplicity-free representations for infinite dimensional cases. 

Let G be a connected, simple non-compact Lie group of Hermitian type. 
We begin with branching laws without continuous spectra, and then discuss 
branching laws with continuous spectra. 

1) (Discretely decomposable case) Let (G, H) be a symmetric pair of holo- 
morphic type. Then, any unitary highest weight representation tt of G decom- 
poses discretely when restricted to H (Fact 5.1). 

1-a) Suppose now that tt is a holomorphic discrete series representation. 
L.-K. Hua [26], B. Kostant, W. Schmid [78] and K. Johnson [32] found an 
explicit formula of the restriction ttJk (if -type formula). This turns out to be 
multiplicity-free. Alternatively, the special case of Theorem B (2) by setting 
H = K gives a new proof of this multiplicity-free property. 

1-b) Furthermore, we consider a generalization of the Hua-Kostant- 
Schmid formula from compact H to noncompact H, for which Theorem B 
(2) still ensures that the generalization will be multiplicity-free. This gener- 
alized formula is stated in Theorem 8.3, which was originally given in [39, 
Theorem C]. In Section 8, we give a full account of its proof. W. Bertram 
and J. Hilgert [7] obtained some special cases independently, and Ben Said 
[5] studied a quantative estimate of this multiplicity-free i/-type formula (see 
also [90, 91] for some singular cases). 

1-c) The branching formulae of the restriction of singular highest weight 
representations tt are also interesting. For instance, the restriction of the 
Segal Shale- Weil representation m of Mp(n, R) with respect to U{p,n — p) 
(more precisely, its double covering) decomposes discretely into a multiplicity- 
free sum of the so called ladder representations of U{p,n — p) (e.g. [33, In- 
troduction]). This multiplicity-free property is a special case of Howe's corre- 
spondence because {U{p,n — p), U (1)) forms a dual pair in Mp{n, R), and also 
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is a special case of Theorem A because (sp(n,]R), u{p,n — p)) forms a symmet- 
ric pair. Explicit branching laws for most of classical cases corresponding to 
Theorems B (2) and D (2) (see Theorems 8.3, 8.4, 8.11) can be obtained by 
using the "see-saw dual pair" , which we hope to report in another paper. 
2) (Branching laws with continuous spectra) Suppose tti is a highest weight 
module and n2 is a lowest weight module, and both being of scalar type. 

2-a) If both TTi and 7r2 are discrete series representations in addition, then 
the tensor product 7ri(E'7r2 is unitarily equivalent to the regular representa- 
tion on L'^{G/K,x), the Hilbert space of L^-sections of the G-equivalent line 
bundle G x^C^ — > G/K associated to some unitary character x oi K (R. 
Howe [23], J. Rcpka [74]). In particular, Theorem C gives a new proof of 
the multiplicity- free property of the Plancherel formula for L^(G/K,x)- Yet 
another proof of the multiplicity-free property of L'^{G/K,x) was given in 
[47, Theorem 21] by still applying Theorem 2.2 to the crown domain (equiv- 
alently, the Akhiezer-Gindikin domain) of the Riemannian symmetric space 
G/K. The explicit decomposition of L'^{G/K,x) was found by J. Heckman 
[20] and N. Shimcno [79] that generalizes the work of Harish-Chandra, S. 
Helgason, and S. Gindikin-F. Karpelevich for the trivial bundle case. 

In contrast to Riemannian symmetric spaces, it is known that "multiplicity- 
free property" in the Plancherel formula fails for (non- Riemannian) symmetric 
spaces G/H in general (see [3, 8] for the description of the multiplicity of the 
most continuous series representations for G/H in terms of Weyl groups). 

2-b) Similarly to the case 2-a), the restriction tt\h for a symmetric pair 
(G, H) of non-holomorphic type is multiplicity-free and is decomposed into 
only continuous spectra if tt is a holomorphic discrete series of scalar type. 
This case was studied by G. Olafsson-B. 0rsted ([69]). 

2-c) Theorem C applied to non-discrete series representations tti and 7r2 
(i.e. tensor products of singular unitary highest weight representations) pro- 
vides new settings of m\iltiplicity-frec branching laws. They might be inter- 
esting from the view point of representation theory because they construct 
"small" representations as discrete summands. (We note that irreducible uni- 
tary representations of reductive Lie groups have not been classified even in 
the spherical case. See [4] for the split case.) They might be interesting also 
from the view point of spectral theory and harmonic analysis which is rele- 
vant to the canonical reprcscnt,ation in the sense of Vcrshik Gelfand -Graev. 
Once we know the branching law is a priori multiplicity-free, it is promising 
to obtain its explicit formula. Some special cases have been worked on in this 
direction so far, for G = 5L(2,R) by V. F. Molchanov [64]; for G = SU{2, 2) 
by B. 0rsted and G. Zhang [70]; for G = SU{n, 1) by G. van Dijk and S. HiUe 
[10]; for G = SU{p,q) by Y. Neretin and G. Ol'shanskii [66, 67]. See also G. 
van Dijk-M. Pevzner [11], M. Pevzner [72] and G. Zhang [92]. Their results 
show that a different family of irreducible unitary representations (sometimes, 
spherical complementary series representations) can occur in the same branch- 
ing laws and each multiplicity is not greater than one. 
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1.9 Organization of this article 

This paper is organized as follows: In Section 2, we give a proof of an abstract 
multiplicity-free theorem (Theorem 2.2) in the line bundle setting. This is an 
extension of a theorem of Faraut-Thomas [15], whose idea may go back to 
Gelfand's proof [17] of the commutativity of the Hecke algebra L^{K\G/K). 
Theorem 2.2 is a main method in this article to find various multiplicity- free 
theorems. In Section 3, wc use Theorem 2.2 to give a proof of Theorem A. The 
key idea is the reduction of the geometric condition (2.2.3) {strongly visible 
action in the sense of [47]) to the existence problem of a "nice" involutive 
automorphism cr of G satisfying a certain rank condition. Section 4 considers 
the multiplicity-free theorem for the tensor product representations of two ir- 
reducible highest (or lowest) weight modules and gives a proof of Theorem C. 
Sections 5 and 6 examine our assumptions in our multiplicity-free theorems 
(Theorems A and C). That is, we drop the assumption of 'scalar type' in 
Section 5 and prove that multiplicities are still uniformly bounded (Theo- 
rems B and D). Wc note that multiplicities can be greater than one in this 
generality. In Section 6, we leave unchanged the assumption that (G, H) is a 
symmetric pair, and relax the assumption that tt is a highest weight module. 
We illustrate by examples a wild behavior of multiplicities without this as- 
sumption. In Section 7, analogous results of Theorems A and C are proved for 
finite dimensional representations of compact groups. In Section 8, we present 
explicit branching laws that are assured a priori to be multiplicity-free by The- 
orems A and C. Theorem 8.4 generalizes the Hua-Kostant-Schmid formula. 
In Section 9 (Appendix) we present some basic results on homogeneous line 
bundles for the convenience of the reader, which give a sufficient condition for 
the assumption (2.2.2) in Theorem 2.2. 

2 Main machinery from complex geometry 

J. Faraut and E. Thomas [15], in the case of trivial twisting parameter, gave 
a sufficient condition for the commutativity of Endij(?i) by using the theory 
of reproducing kernels, which we extend to the general, twisted case in this 
preliminary section. The proof parallels to theirs, except that we need just 
find an additional condition (2.2.2) when we formalize Theorem 2.2 in the 
line bundle setting. 

2.1 Basic operations on holomorphic line bundles 

Let C ^ D he a. holomorphic line bundle over a complex manifold D. We 
denote by 0{C) = 0{D, C) the space of holomorphic sections oiC ^ D. Then 
0{C) carries a Frechet topology by the uniform convergence on compact sets. 
If a Lie group H acts holomorphically and equivariantly on the holomorphic 
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line bundle C ^ D, then H defines a (continuous) representation on 0{C) by 
the pull-back of sections. 

Let {Ua} be trivializing neighborhoods of D, and gajj G {Ua n Ujs) the 
transition functions of the holomorphic line bundle C ^ D. Then an anti- 

holomorphic line bundle £ ^ _D is a complex line bundle with the transition 
functions 'g^. We denote by 0{C) the space of anti- holomorphic sections for 

Suppose a is an anti-holomorphic diffcomorphism of D. Then the pull-back 
a*C ^ £) is an anti-holomorphic line bundle over D. In turn, cr*£ ^ f is a 
holomorphic line bundle over D (see Appendix for more details). 

2.2 Abstract multiplicity-free theorem 

Hero is the main machinery to prove various multiplicity-free theorems of 
branching laws including Theorems A and C (infinite dimensional represen- 
tations) and Theorems E and F (finite dimensional representations). 

Theorem 2.2. Let {n,Tl) be a unitary representation of a Lie group H. As- 
sume that there exist an H-equivariant holomorphic line bundle C ^ D and 
an anti-holomorphic involutive diffcomorphism a of D with the following three 
conditions: 

(2.2.1) There is an infective (continuous) H -intertwining map H — > 0{jC). 

(2.2.2) There exists an isomorphism of H-equivariant holomorphic line bun- 
dles If : £ ^ a*Z. 

(2.2.3) Given x G D, there exists g G H such that a{x) = g ■ x. 

Then, the ring End^f(7i) of continuous H -intertwining operators on Ti is 
commutative. Consequently, (7r,7i) is multiplicity-free (see Definition 1.1). 

2.3 Remarks on Theorem 2.2 

This subsection gives brief comments on Theorem 2.2. First, we consider a 
special case, and also a generalization. 

Remark 2.3.1 (specialization and generalization). 1) Suppose £ — > is the 
trivial line bundle. Then, the condition (2.2.2) is automatically satisfied. In 
this case. Theorem 2.2 was proved in [15]. 

2) An extension of Theorem 2.2 to the equivariant vector bundle V — > .D 

is the main subject of [49], where a more general multiplicity-free theorem 
is obtained under an additional condition that the isotropy representation of 
Hx = {h G H : h - X = x} on the fiber Vx is multiplicity- free for generic x G D. 
Obviously, the iJj;-action on Vx is multiplicity-free for the case dimVx = 1, 
namely, for the line bundle case. 

Next, we examine the conditions (2.2.2) and (2.2.3). 
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Remark 2.3.2. 1) In many cases, the condition (2.2.2) is naturally satisfied. 
Wc shall explicate how to construct the bundle isomorphism in Lemma 9.4 
for a Hermitian symmetric space D. 

2) As the proof below shows, Theorem 2.2 still holds if we replace D by an 
iJ-invariant open subset D' . Thus, the condition (2.2.3) is local. The concept 
of 'visible action' (see [46, 49, 51]) arises from the condition (2.2.3) on the 
base space D. 

3) The condition (2.2.3) is automatically satisfied if H acts transitively on 
D. But we are interested in a more general setting where each iJ-orbit has 
a positive codimension in D. We find in Lemma 3.3 a sufficient condition for 
(2.2.3) in terms of rank condition for a symmetric space D. 

2.4 Reproducing kernel 

This subsection gives a quick summary for the reproducing kernel of a Hilbert 
space H. realized in the space 0{C) of holomorphic sections for a holomorphic 
line biuidle C (see [49] for a generalization to the vector bundle case). Since 
the reproducing kernel K-^ contains all the information on the Hilbert space 
H, our strategy is to make use of K-h in order to prove Theorem 2.2. 

Suppose that there is an injectivc and continuous map for a Hilbert space 
7i into the Frechet space 0{C). Then, the point evaluation map 

o(/:)dh^/:.^c, f^f{z) 

is continuous with respect to the Hilbert topology on Ji. 
Let {lyJj/} be an orthonormal basis of H. We define 

Kn{x, y) = K{x, y) := ^ M^YMy) & 0{C)®0{C) . 

Then, K{x,y) is well-defined as a holomorphic section of £ ^ D for the 
first variable, and as an anti-holomorphic section of £ — > £) for the second 
variable. The definition is independent of the choice of an orthonormal basis 
{^u}- K(x, y) is called the reproducing kernel of ?i. 

Lemma 2.4. 1) For each y G D, K{-,y) G n®C^ (~ H) and {f{-),K{-, y))u = 
f{y) for any f gH. 

2) Let Ki{x,y) he the reproducing kernels of Hilbert spaces Hi C 0{C) with 
inner products ( , respectively, for i = 1,2. If Ki = K2, then Hi = H2 
and ( , )wi = ( , )h2- 

3) If Ki{x,x) = K2{x,x) for any x G D, then Ki = K2. 

Proof. (1) and (2) are standard. We review only the way how to recover Ti. 
together with its inner product from a given reproducing kernel. For each 
y e r>, we fix an isomorphism >Cy ~ C. Through this isomorphism, we can 
regard K{-,y) G H, ® Cy as an clement of Ji. The Hilbert space TL is the 
completion of the C-span of {K{-,y) : y G D} with pre-Hilbert structure 
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{K{-,yi), K{-,y2))n ■= K{y^, y^) e Cy, ® Cy, (~ C) . 



(2.4.1) 



This procedure is independent of the choice of the isomorphism £y ~ C. 
Hence, the Hilbert space H together with its inner product is recovered. 
3) Wc denote by D the complex manifold endowed with the conjugate 
complex structure on D. Then, £ ^ D is a holomorphic line bundle, and 
K{-,-) = K-h{-,-) is a holomorphic section of the holomorphic line bundle 
£ £ — !■ D X D. As the diagonal embedding l : D ^ D x D , z ^ {z, z) \s 
totally real, {Kx — K2)\i,(d) = implies Ki — K2 = Q hy the unicity theorem 
of holomorphic functions. □ 

2.5 Construction of J 

Suppose we are in the setting of Theorem 2.2. We define an anti-linear map 



by Jf{z) := f{a{z)) {z G D). Jf is regarded as an element of ©(£) through 
the isomorphism : 0{C) ~ 0{a*C) (see (2.2.2)). 

Lemma 2.5. In the setting of Theorem 2.2, we identify H with a subspace of 
0{C). Then, the anti-linear map J is an isometry from H onto H. 

Proof. We put H := J{T-L), equipped with the inner product 



If {ipu} is an orthonormal basis of 7Y, then H is a Hilbert space with or- 
thonormal basis {J(py}. Hence, the reproducing kernel of Ti is given by 
Kj^{x,y) = K'n{a{y),a{x)) because 



Kf^{x,y) =^J(f^{x)J(p^{y) = ^ip^{a{x)) ^Pu{a{y)) = Kn{cr{y), a{x)) . 

(2.5.2) 



We fix a; e and take g & H such that a{x) — g-x (see (2.2.3)). Substituting 
X for y in (2.5.2), we have 

Kf^{x,x) = Ku{cr{x),a{x)) = Kn{g ■ x, g ■ x) = Ku{x,x) . 

Here, the last equality holds because {Vi/{g ■)} is also an orthonormal basis 
of H as {tTjH) is a unitary representation of H. Then, by Lemma 2.4, the 
Hilbert space H coincides with H and 



J:0(£)^0(£), f^Jf 



{Jfi , Jf2)n ■=if2, .fi )n for /i , /a € H . 



(2.5.1) 



{Jfi,Jf2)n = (/2, fi)n for /i, /2 e W . 



(2.5.3) 



This is what we wanted to prove. □ 
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2.6 Proof of A* = JAJ-^ 

Lemma 2.6 (see [15]). Suppose A E End^f(?i). Then the adjoint operator A* 
of A is given by 

A* = JAJ-^ . (2.6.1) 
Proof. We divide the proof into three steps. 

Step 1 (positive self-adjoint case): Assume A G End//(H) is a positive self- 
adjoint operator. Let Ha be the Hilbert completion of H by the pre-Hilbert 
structure 

(.fi,/2)H^ :=(M,/2)h for/i,/2GH. (2.6.2) 

If /i, /2 e and g e H, then 

(7i-(ff)/i,7r(5()/2)HA = {^'^{9)fi,T^i9)f2)n 

= (Tr{g)Afi,TT{g)f2)n = {Afi,f2)n = (,/i,./2)-Ha • 

Therefore, (tt, H) extends to a unitary representation on Ha- Applying (2.5.3) 
to both Ha and H, we have 

{Afuf2)H = (,/i,./2)w^ = iJf2,Jfi)n^ = {AJf2,Jfi)n 

= iJf2,A*Jfi)n = {Jf2,JJ-'A*Jfi)H = iJ-'A*Jfij2)H- 

Hence, A = J^^A*J, and (2.6.1) follows. 

Step 2 (self-adjoint case): Assume A e Endjj(W) is a self-adjoint operator. 

Let ^ = / XdEx be the spectral decomposition of A. Then every projection 
operator Ex € End(?i) also commutes with 7r((?) for all g ^ H, namely, 
Ex e EndH(W). We define 

A+ := / XdEx , A_ := / Adi;^ . 

Then ^ = A^ + A-. Let / be the identity operator on H. As a positive self- 
adjoint operator A+ + / is an element of End// (7Y), we have {A-^- + I)* = 
J(A+ + I)J~^ by Step 1, whence AZ^ — JAj^J^^. Applying Step 1 again to 
we have A*_ = JA^J'^. Thus, 

A* =Al + A*_= JA+J-^ + J A.J-'' = J{A+ + A_)J-^ = JAJ'^ . 

Step 3 (general case): Suppose A e Endi/(H). Then A* also commutes with 
^(5) {9 S H) because tt is unitary. We put B := ^{A + A*) and C := 
■^^^^(A* — A). Then, both B and C are self-adjoint operators commuting with 
iT{g) {g e H). It follows from Step 2 that = JBJ"! and C* = JCJ-\ As 
J is an anti-linear map, we have 

Hence, A = B + ^/^C also satisfies A* = JAJ-^. □ 
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2.7 Proof of Theorem 2.2 



We are now ready to complete the proof of Theorem 2.2. Let A, B e End_f/(H). 
By Lemma 2.6, we have 

AB = J^^AByj = {J-'^B*J){J-'^A*J) = BA. 

Therefore, Endi/(H) is commutative. □ 

3 Proof of Theorem A 

This section gives a proof of Theorem A by using Theorem 2.2. The core of the 
proof is to reduce the geometric condition (2.2.3) to an algebraic condition 
(the existence of a certain involution of the Lie algebra). This reduction is 
stated in Lemma 3.3. The reader who is familiar with symmetric pairs can 
skip Subsections 3.1, 3.2, 3.4 and 3.5. 

3.1 Reductive symmetric pairs 

Let G be a Lie group. Suppose that r is an involutive automorphism of G. 
We write 

:= {.9 e G : rg = g} 

for the fixed point subgroup of r, and denote by GJ its connected component 
containing the unit element. The pair (G, H) (or the pair {q, I)) of their Lie 
algebras) is called a symmetric pair if the subgroup H is an open subgroup 
of G'^, that is, a H satisfies 

GlcHc G\ 

It is called a reductive symmetric pair if G is a reductive Lie group; a semisim- 
ple symmetric pair if G is a semisimple Lie group. Obviously, a semisimple 
symmetric pair is a reductive symmetric pair. 

We shall use the same letter r to denote the differential of r. We set 

.= {y g g . = ±Y} . 
Then, it follows from = id that we have a direct sum decomposition 

Suppose now that G is a semisimple Lie group. It is known that there 
exists a Cartan involution 6* of G commuting with r. Take such 6, and we 
write K :— G^ — {g ^ G : 9g = g}. Then, K is compact if G is a linear Lie 
group. The direct sum decomposition 

fl = e © p = 0^ © £|-^ 
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is called a Cartan decomposition. Later, we shall allow G to be non-linear, 

in particular. K is not necessarily compact. The real rank of g, denoted by 
R- rank 9, is defined to be the dimension of a maximal abelian subspace of 

As {t6)^ = id, the pair {q, q^^) also forms a symmetric pair. The Lie group 

G^' = {g€G: {Te){g) = g] 

is a reductive Lie group with Cartan involution 6\qt6, and its Lie algebra 
is reductive with Cartan decomposition 

Here, we have used the notation and alike, defined as follows: 

Then, the dimension of a maximal abelian subspace a of is equal to 

the real rank of , which is referred to as the split rank of the semisimple 
symmetric space G/H. We shall write M-rankG/if or R-rankg/g'^ for this 
dimension. Thus, 

R-rankg^^ = R-rankg/g^. (3.1.2) 

In particular, we have R-rankg = R-rankg/8 if we take t to be 6. 

The Killing form on the Lie algebra g is non-degenerate on g, and is also 
non-degenerate when restricted to f). Then, it induces an Ad(ff)-invariant 
non-degenerate bilinear form on g/f), and therefore a G- invariant pseudo- 
Riemannian structure on the homogeneous space G/H , so that G/H becomes 
a symmetric space with respect to the Levi-Civita connection and is called 
a semisimple symmetric space. In this context, the subspace a has the fol- 
lowing geometric meaning: Let A := exp(o), the connected abelian subgroup 
of G with Lie algebra a. Then, the orbit A ■ o through o := eH S G/H be- 
comes a flat, totally geodesic submanifold in G/H. Furthermore, we have a 
(generalized) Cartan decomposition: 

Fact 3.1 (see [16, Section 2]). G = KAH. 

Sketch of Proof. The direct sum decomposition of the Lie algebra 

g = e©g-^.-«©g^.-« 

lifts to a diffeomorphism: 

g-"'-' + g"'-'^i^\G, (X,y)^Ke^e^. 

Since exp(g'^'~^) c H, the decomposition G = KAH follows if we show 

Ad{HnK)a = g-'''-^. (3.1.3) 

The equation (3.1.3) is well-known as the key ingredient of the original Cartan 
decomposition G^" = K'^AK'' in light of (3.1.1). □ 
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Furthermore, suppose that a is an involutive automorphism of G such that 
cr, r and 6 commute with one another. We set 

G-'^ ■=G''f^G- = {g&G:ag = Tg = g}. 

Then {G'^ ,G'^''^) forms a reductive symmetric pair, because a and r commute. 

The commiitativity of cr and 9 implies that the automorphism a : G ^ G 
stabihzes K and induces a diffeomorphism of G/K, for which we use the 
same letter <j. 

3.2 Ex£imples of symmetric pairs 

This subsection presents some basic examples of semisimple (and therefore, 

reductive) symmetric pairs. 

Example 3.2.1 (group manifold). Let G' be a semisimple Lie group, and G := 
G' X G'. We define an involutive automorphism r of G by T{x,y) := {y,x). 
Then, G'^ = {{g,g) ■ g G G'} is the diagonal subgroup, denoted by diag(G"), 
which is isomorphic to G' . Thus, (G" x G",diag(G")) forms a semisimple sym- 
metric pair. 
We set 




v-4 / 

Example 3.2.2. Let G = SL{n, C), and fix p, q such that p + q = n. Then, 

r{g) ■■= lp,q g* lp,q {g^G) 

defines an involutive aTitomorphism of G, and G"^ = SU (p, q) (the indefi- 
nite unitary group). Thus, {SL{n, C), SU(j>, q)) forms a semisimple symmetric 
pair. 

Example 3.2.3. Let G = SL{n,C), and (T{g) := g. Then a is an involutive 

automorphism of G, and G'^ = SL{n,M.). We note that a commutes with the 
involution r in the previous example, and 

G^'^ = {g e SL{n, C):g = g = Ip,g^ Ip,g} 
= SO{p,q). 
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Thus, {SL{n,C),SL{n,R)), {SU{p,q), SO{p,q)), {SL{n,R), SO{p,q)) are ex- 
amples of semisimple symmetric pairs. 

Example 3.2.1 Let G := 5L(2n,R), and T{g) := J*g~^J-\ Then, = 
Sp{n,R) (the real symplectic group). Thus, {SL{2n,M.), Sp{n,M.)) forms a 
semisimple symmetric pair. 

3.3 Reduction of visibility to real rank condition 

The following lemma gives a sufficient condition for (2.2.3). Then, it plays 
a key role when we apply Theorem 2.2 to the branching problem for the 
restriction from G to G"^ (with the notation of Theorem 2.2, D = G/K and 
H = Gq). This lemma is also used in reducing 'visibility' of an action to an 
algebraic condition ([50, Lemma 2.2]). 

Lemma 3.3. Let a and t he involutive automorphisms of G. We assume that 
the pair (ct, t) satisfi,es the following two conditions: 

(3.3.1) a, T and 6 commute with one another. 

(3.3.2) M-rankfl'"^ = R-rankg'^'^^. 

Then for any x G G/K, there exists g & Gq such that a{x) = g ■ x. 

Proof. It follows from the condition (3.3.1) that 9\a^ is a Cartan involution 
of a reductive Lie group G"^ and that t|g^ is an involutive automorphism of 
G"^ commuting with Olc ■ Take a maximal abelian subspace o in 

Q-0,T,re ._^Y€2: {-0)Y = aY = tOY = Y} . 

From definition, we have dim o = M- rankfl'^''^^, which in turn equals R- rankfl"^^ 
by the condition (3.3.2). This means that a is also a maximal abelian subspace 
in 

^-e,re ^ |y g g . ^ ^gy = Y} . 

Let A = exp(a). Then it follows from Fact 3.1 that we have a generalized 
Cartan decomposition 

G = GIAK . (3.3.3) 

Let o := eK e G/K. Fix x € G/K. Then, according to the decomposition 

(3.3.3) , we find h & Gq and a G A such that 

X = ha- 0. 

We set g := a{h) h~^. We claim g G Gq. In fact, by using ctt = tct and 
rh = h, we have 

T{g) = Tcr{h) T{h-^) = aT{h) t(/i)-^ = a{h) h'^ = g . 

Hence, g G G'^. Moreover, since the image of the continuous map 
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is connected, we have g G Gq. 

On the other hand, we have a{a) = a because o C q~^''^'~'^ c q'^. Therefore 
we have 

a{x) = <j{h) a{a) ■ o = (j{h) h~^ha ■ o = g ■ x , 
proving the lemma. □ 

3.4 Hermitian Symmetric Space G/K 

Throughout the rest of this section, we assume that G is a simple, non- 
compact, Lie group of Hermitian type. We retain the notation of Subsec- 
tion 1.3. 

Let Gc be a connected complex Lie group with Lie algebra Qcj and Q~ the 
maximal parabolic subgroup of Gc with Lie algebra fic + P-- Then we have 
an open embedding G/K '-^ Gc/Q~ because Qc = + (fic + P-)- Thus, a 
G-invariant complex structure on G/K is induced from Gc/Q~ ■ (We remark 
that the embedding G/K ^ Gc/Q~ is well-defined, even though G is not 
necessarily a subgroup of Gc-) 

Suppose r is an involutive automorphism of G commuting with 6. We 
recall from Subsection 1.4 that we have either 

tZ = Z (holomorphic type), 

or 

tZ = —Z (anti-holomorphic type). (1-4-2) 

Here is the classification of semisimple symmetric pairs (0,0"^) with simple 

such that the pair (0,0'^) satisfies the condition (1.4.1) (respectively, (1.4.2)). 
Table 3.4.2 is equivalent to the classification of totally real symmetric spaces 
G'^/K'' of the Hermitian symmetric space G/K (see [14, 27, 28, 35]). 
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Table 3.4.1. 



(0,0'') is of holomorphic type 







su{p, q) 


s{u{i,j)+u{p-i,q-j)) 


su{n,n) 


so*(2n) 


su(n, n) 


sp(n,K) 


so*(2n) 


so*(2p) +so*(2n - 2p) 


so*(2n) 


u(p, n-p) 


so(2, n) 


so{2,p) +so{n -p) 


so(2,2n) 


u(l,n) 


sp(n,K) 


u(p, n-p) 


sp(n,]R) 


5p(p,K) +sp(n -p,K) 


e6{-i4) 


SO(IO) +50(2) 


66(-14) 


so*(10) +so(2) 


66{-14) 


so(8,2) +so(2) 


e6(-i4) 


su(5, l)+s[(2,K) 


C6(-14) 


su(4, 2) + su(2) 


C7{-25) 


e6(_78) +so(2) 


C7{-25) 


e6(-i4) +so(2) 


«7(-25) 


so(10,2) +s[(2,K) 


e7(-25) 


so*(12) +su(2) 


e7(-25) 


su(6,2) 



Table 3.4.2. 



(0,0^) is of anti-holomorphic type 





0" 


su(p, (?) 


so(p,g) 


su(n, n) 


sl(n,C) +E 


su(2p, 2g) 




so*(2n) 


so(n, C) 


so*(4n) 


BU*(2n) +E 


so(2, n) 


so(l,p) +so(l,n -p) 


5p(n, E) 


0l(n,R) 


5p(2n,E) 


sp(n,C) 


f:6(-14) 


f4{-20) 


e6(-14) 


sp(2,2) 


e7(-25) 


e6(-26) +so(l, 1) 


C7(-25) 


BU*(8) 
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3.5 Holomorphic realization of highest weight representations 

It is well-known that an irreducible highest weight representation tt of G can 
be realized as a subrepresentation of the space of global holomorphic sections 
of an cquivariant holomorphic vector bundle over the Hcrmitian symmetric 
space G/K. We supply a proof here for the convenience of the reader in a way 
that we shall use later. 

Lemma 3.5. Let (tt, Ti) he an irreducible unitary highest weight module. We 
write X for the representation of K on U := H'^j^ (see Definition 1.3). Let C := 
GxkU ^ G/K be the G-equivariant holomorphic vector bundle associated to 
X- Then, there is a natural injective continuous G-homomorphism H — » 0{£). 

Proof. Let ( , be a G-invariant inner product on H. We write ( , )u for the 
induced inner product on U. Then, K acts unitarily on H, and in particular 
on U. We consider the map 

G xHxU ^C, {g,v,u)^ {nigy^v, u)n = {v, ■K{g)u)u ■ 

For each fixed g G G and v G H, the map C/ — > C, u i— > {Tr{g)~^v,u)'n is an 
anti- linear functional on U. Then there exists a unique clement Fy{g) G C/ by 
the Riesz representation theorem for the finite dimensional Hilbert space U 
such that 

{Fy{g),u)u = {Tr{g)~'^v,u)n for any u e C/. 

Then it is readily seen that Fy{gk) = x{k)~^Pv{g) and Fj^(gi-fy{g) = Fy{g' ^g) 
for any g,g' G G, k G K and v G H. As u is a. smooth vector in H, 
{Fv{g),u)u = {v,TT{g)u)H is a G°°-function on G. Then Fy{g) is a C°°- 
function on G with value in U for each fixed v G H. Thus, we have a non-zero 
G- intertwining operator given by 

F -.H^C^iGxKU), v^Fy. 

As U is annihilated by p+, Fy is a holomorphic section of the holomorphic 
vector bundle G xkU ^ G/K, that is, Fy G 0{G xkU). Then, the non-zero 
map F -.H ^ 0(G xk U) is injective because H is irreducible. Furthermore, 
F is continuous by the closed graph theorem. Hence, Lemma 3.6 is proved. 
□ 

3.6 Reduction to real rank condition 

The next Lemma is a stepping-stone to Theorem A. It becomes also a key 
lemma to the theorem that the action of a subgroup H on the bounded sym- 
metric domain G/K is 'strongly visible' for any symmetric pair {G,H) (see 
[50]). 
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Lemma 3.6. Suppose g is a real simple Lie algebra of Hermitian type. Letr be 

an involutive automorphism of q, commuting with a fixed Cartan involution 9. 
Then there exists an involutive automorphism a of Q satisfying the following 
three conditions: 

(3.6.1) a, T and 9 commute with one another. 

(3.6.2) R-rankfl^^ = R-rankg'"''"^. 

(3.6.3) crZ = -Z. 

Proof. We shall give a proof in the special case t = 9 in Subsection 4.1. For 
the general case, see [50, Lemma 3.1] or [44, Lemma 5.1]. □ 

3.7 Proof of Theorem A 

Now, we arc ready to complete the proof of Theorem A. 

Without loss of generality, we may and do assume that G is simply con- 
nected. Let (tt, T-L) be an irreducible unitary highest weight representation of 
scalar type. Wc define a holomorphic line bundle by £ := G Ti^f^ over the 
Hermitian symmetric space D := G/ K. Then, it follows from Lemma 3.5 that 
there is an injective continuous G-intertwining map H — > 0{C). 

Suppose (G, H) is a symmetric pair. We first note that for an involutive 
automorphism r of G, there exists 5 G G such that t^9 = 9r^ if we set 

T^{x) := gT{g~^xg)g~^ 

for X & G. Then, G'^f = gHg~^ is 0-stable. Since the multiplicity-free property 
of the restriction Trjij is unchanged if we replace H by gHg~^, we may and 
do assume that 9H = H, in other words, 9t = t9. 

Now, by applying Lemma 3.6, we can take a satisfying (3.6.1), (3.6.2) 
and (3.6.3). We use the same letter a to denote its lift to G. It follows from 
(3.6.3) that the induced involutive diffeomorphism a : G/K ^ G/K is anti- 
holomorphic (see Subsection 1.4). In light of the conditions (3.6.1) and (3.6.2), 
we can apply Lemma 3.3 to see that for any x & D there exists g & H such 
that a{x) = g ■ X. 

Moreover, by using Lemma 9.4 in the Appendix, we have an isomorphism 
a*C ~ £ as G-equivariant holomorphic line bundles over G/K. Therefore, 
all the assumptions of Theorem 2.2 are satisfied. Thus, we conclude that the 
restriction Trjij is multiplicity- free by Theorem 2.2. □ 



4 Proof of Theorem C 

In this section we give a proof of Theorem C. 

Throughout this section, wc may and do assume that G is simply connected 
so that any automorphism of Q lifts to G. We divide the proof of Theorem C 
into the following cases: 
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Case I. Both tti and 7r2 are highest weight modules. 
Case I'. Both tti and tt2 are lowest weight modules. 

Case II. TTi is a highest weight module, and 1^2 is a lowest weight module. 
Case II'. TTi is a lowest weight module, and 772 is a highest weight module. 

4.1 Reduction to real rank condition 

The following lemma is a special case of Lemma 3.6 with t = 0. We shall see 
that Theorem C in Case I (likewise. Case I') reduces to this algebraic result. 

Lemma 4.1.1. Suppose q is a real simple Lie algebra of Hermitian type. Let 
be a Cartan involution. Then there exists an involutive automorphism a of 
Q satisfying the following three conditions: 

(4.1.1) a and 6 commute. 

(4.1.2) R- rank = R- rank fl"^. 

(4.1.3) crZ = -Z. 

Proof. We give a proof of the Lemma based on the classification of simple Lie 
algebras g of Hermitian type. 

We recall that for any involutive automorphism cr of G, there exists g € G 
such that (t'-'O = 9a^ . Thus, (4.1.1) is always satisfied after replacing a by 
some . The remaining conditions (4.1.2) and (4.1.3) (cf. Table 3.4.2) are 
satisfied if we choose cr e Aut(G) in the following Table 4.1.2 for each simple 
non-compact Lie group G of Hermitian type: 



Table 4.1.2. 



(0,5") satisfying (4.1.2) and (4.1.3) 


fl 


0" 


R-rankfl = R- rank 5°" 


su(p, q) 


so(p,g) 


min(p, q) 


50* (2n) 


so{n, C) 




sp(n,R) 


fl[(n,K) 


n 


50(2, n) 


so(l,n - 1) +so(l, 1) 


min(2, n) 


e6{-i4) 


5p(2,2) 


2 


«7(-2B) 


su*(8) 


3 



Here, we have proved Lemma. □ 

Remark 4-1-3. The choice of a in Lemma 4.1.1 is not unique. For example, 
we may choose fl'^ ~ ^6(-26) ® ^ instead of the above choice g"^ ~ su*(8) for 

= e7(-25)- 
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4.2 Proof of Theorem C in Case I 

Let G be a non-compact simply-connected, simple Lie group such that G/ K 
is a Hcrmitian symmetric space. 

Let (TTijTii) and [112, be two irreducible imitary highest weight rep- 
resentations of scalar type. By Lemma 3.5, we can realize {7Ti,7ii) in the 
space 0{Ci) of holomorphic sections of the holomorphic line bundle Ci := 
G Xk {'HiYj^ {i = 1,2) over the Hermitian symmetric space G/K. We now 
define a holomorphic line bundle C := Ci Kl £2 over D := G/K x G/K as 
the outer tensor product of £1 and €2- Then, we have naturally an injective 
continuous (G x G)-intertwining map Tli$}'Tl2 C>{C). 

Let us take an involution a' of g as in Lemma 4.1.1 (but we use the letter a' 
instead of a), and lift it to G. We set a := a' x a'. Then it follows from (4.1.3) 
that a' acts anti-holomorphically on G/K, and so does a on D. Furthermore, 
we have isomorphisms of holomorphic line bundles {a')* Ci Ci (i = 1, 2) by 
Lemma 9.4 and thus a* Cc:± C. 

We now introduce another involutive automorphism t of G x G by 
r(.9i,.92) := (52,51)- Then (G x GY = diag(G) := {(,9,, 9) : ,9 e G}. We shaU 
use the same letter 9 to denote the Cartan involution 9 x 6 on G x G (and 
9 (B on Q(B &)■ Then, we observe the following isomorphisms: 

{Q®Qy' = {{X,eX):XGg} 

(0 e qY'^^ = { (X, ^x) : x e 0^' } ~ . 

Thus, the condition (4.1.2) implies 

]R-rank(0 = M-rank(0 0)"''"^ . 

Therefore, given (xi,X2) e D ~ (Gx G)/ (A' x A'), there exists {g,g) G {GxGY 
satisfying (9 • xi,g ■ X2) = {(j' {xi),(j' {X2)) (= a{xi,X2)) by Lemma 3.3. 

Let us apply Theorem 2.2 to the setting {C D, Tii(^Ti.2, diag(G), cr). Now 
that all the assumptions of Theorem 2.2 are satisfied, we conclude that the 
tensor product 7ri®7r2 is multiplicity-free as a G-module, that is. Theorem C 
holds in the case L □ 

4.3 Proof of Theorem C in Case II 

Let us give a proof of Theorem C in the case IL We use the same r as in 

Subsection 4.2, that is, T{gi,g2) '■= (.92,. 9i) and define a new involution a by 
a := t9, that is, cr{gi,g2) = {9g2,9g\) for 51,52 € G. Obviously, cr, r and the 
Cartan involution ^ of G x G all commute. 

We write M for the Hermitian symmetric space G/K, and M for the 
conjugate complex manifold. Then a acts anti-holomorphically onD := M x 
M because so does r and because 9 acts holomorphically. 

By the obvious identity (b®0)^^ = (0®fl)'^-^^: we have M- rank(0 ® g)^^ = 
R-rank(0 © 0)'^''^^ (= M-rank0). Therefore, it follows from Lemma 3.3 that 
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for any {xi,X2) € D there exists {g,g) G {G x GY such that a{xi,x2) = 
(9,9) ■ {X1,X2). 

Suppose TTi (respectively, 772) is a unitary highest (respectively, low- 
est) weight representation of scalar type. We set Ci := G xk (Wi)if and 
£2 '■— G Xk (^2)/^- Then, Ci ^ M and £2 M arc both holomorphic 
line bundles, and we can realize tti in 0(M, £1), and tt2 in C'(M,£2), respec- 
tively. Therefore, the outer tensor product tti K1 772 is realized in a subspace 
of holomorphic sections of the holomorphic line bundle £ := £1 Kl £2 over 
D = M xM. 

Now, we apply Theorem 2.2 to (£ — > D,H\^'H2,dia,g{G),a). The condi- 
tion (2.2.2) holds by Lemma 9.4. Hence, all the assumptions of Theorem 2.2 
are satisfied, and therefore. Theorem C holds in the case II. □ 

Hence, Theorem C has been proved. 

5 Uniformly bounded multiplicities — Proof of 
Theorems B and D 

This section gives the proof of Theorems B and D. Since the proof of Theo- 
rem B parallels to that of Theorem D, we deal mostly with Theorem D here. 
Without loss of generality, we assume G is a non-compact simple Lie group 
of Hermitian type. 

5.1 General theory of restriction 

A unitary representation (7r,W) of a group L is discretely decomposable if 
TT is unitarily equivalent to the discrete Hilbert sum of irreducible unitary 
representations of L: 



Furthermore, we say n is L-admissible ([38]) if all the multiplicities m„{^) are 
finite. In this definition, we do not require m.^{iJ,) to be uniformly bounded 
with respect to /x. 

Suppose L' is a subgroup of L. Then, the restriction of tt to L' is regarded 
as a unitary representation of L'. If tt is i'-admissible, then tt is i-admissible 
([38, Theorem 1.2]). 

We start with recalling from [42] a discrete decomposability theorem of 
branching laws in the following settings: 

Fact 5.1. 1) Suppose t is of holomorphic type (see Definition I.4) and set 
H := Gq. 1} -K is an irreducible unitary highest weight representation of G, 
then TT is (HnK) -admissible. In particular, it is H -admissible. The restriction 
ttIh splits into a discrete Hilbert sum of irreducible unitary highest weight 
representations of H: 
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ttIh — TnaTr{ij)iJL (discrete Hilbert sum), (5.1.1) 
where the multiplicity mT^{ii) is finite for every ji. 

2) Let 7ri,7r2 be two irreducible unitary highest weight representations of G. 
Then the tensor product tti (8)772 is K -admissible under the diagonal action. 
Furthermore, 7ri(8)7r2 splits into a discrete Hilbert sum of irreducible unitary 
highest weight representations of G, each occurring with finite multiplicity. 
Furthermore, if at least one of tti or -K2 is a holomorphic discrete series rep- 
resentation for G, then any irreducible summand is a holomorphic discrete 
series representation. 

Proof. See [42, Theorem 7.4] for the proof. The main idea of the proof is taking 
normal derivatives of holomorphic sections, which goes back to S. Martens [63]. 
The same idea was also employed in a number of papers including Lipsman 
([60, Theorem 4.2]) and Jakobsen-Vergne ([31, Corollary 2.3]). □ 

Remark 5.1. Fact 5.1 (1) holds more generally for a closed subgroup H satis- 
fying the following two conditions: 

1) H is 6'-stablc. 

2) The Lie algebra I] of contains Z. 

Here, we recall that Z is the generator of the center of 6. The proof is essentially 
the same as that of Fact 5.1 (1). 

Theorem B (2) follows from Theorem A and Fact 5.1 (1). Likewise, The- 
orem D (2) follows from Theorem C and Fact 5.1 (2). What remains to show 
for Theorems B and D is the uniform boundedness of multiplicities. 

5.2 RemEirks on Fact 5.1 

Some remarks on Fact 5.1 are in order. 

Remark 5.2.1. A Cartan involution 9 is clearly of holomorphic type because 
9Z = Z. U 6 = T then H = K and any irreducible summand /i is finite 
dimensional. In this case, the finiteness of mT^{ix) in Fact 5.1 (1) is a special 
case of Harish-Chandra's admissibility theorem (this holds for any irreducible 
unitary representation tt of G). 

Rem,ark 5.2.2. Fact 5.1 asserts in particular that there is no continuous spec- 
trum in the irreducible decomposition formula. The crucial assumption for 
this is that {G,H) is of holomorphic type. In contrast, the restriction 'k\h is 
not discretely decomposable if (G, H) is of anti-holomorphic type and if tt is 
a holomorphic discrete series representation of G ([38, Theorem 5.3]). In this 
setting, R. Howe, J. Repka, G. Olafsson, B. 0rsted, van Dijk, S. Hille, M. 
Pevzner, V. Molchanov, Y. Neretin, G. Zhang and others studied irreducible 
decompositions of the restriction Trj// by means of the L^-harmonic analysis 
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on Riemannian symmetric spaces H/HtlK ([9, 10, 11, 23, 64, 66, 69, 70, 74]). 
The key idea in Howe and Repka [23, 74] is that a holomorphic function on 
G/K is uniquely determined by its restriction to the totally real submanifold 
H/H n K (essentially, the unicity theorem of holomorphic functions), and 
that any function on H/ H D K can be approximated (in a sense) by holomor- 
phic functions on G/K (essentially, the Weierstrass polynomial approximation 
theorem) . 

Remark 5.2.3. A finite multiplicity theorem of the branching law (5.1.1) with 
respect to semisimple symmetric pairs (G, H) holds for more general tt (i.e. 
TT is not a highest weight module), under the assumption that tt is discretely 
decomposable as an (f)c, -ff H if )-module (see [41, Corollary 4.3], [45]). How- 
ever, the multiplicity of the branching law can be infinite if the restriction is 
not discretely decomposable (see Example 6.3). 

Rem,ark 5.2.4. Theorems B and D assert that multiplicities m^{fi) in Fact 5.1 
are uniformly bounded when we vary /U. This is a distinguished feature for 
the restriction of highest weight representations tt. A similar statement may 
fail if TT is not a highest weight module (see Example 6.2). 

5.3 Reduction to the scalar type case 

In order to deduce Theorem D (1) from Theorem D (2), we use the idea of 
'coherent family' of representations of reductive Lie groups (for example, see 
[85]). For this, we prepare the following Lemma 5.3 and Proposition 5.4.1. 

Lemma 5.3. Suppose that (tt, H) is an irreducible unitary highest weight rep- 
resentation of G. Then there exist an irreducible unitary highest weight rep- 
resentation tt' of scalar type and a finite dimensional representation F of G 
such that the underlying {qc,K) -module ttk occurs as a subquotient of the 
tensor product w'j^ F. 

Proof. Without loss of generality, we may and do assume that G is simply 
connected. Since G is a simple Lie group of Hermitian type, the center c{t) of 
t is one dimensional. We take its generator Z as in Subsection 1.4, and write 
G for the connected subgroup with Lie algebra c{t). Then, K is isomorphic to 
the direct product group of G and a semisimple group K' . 

As {tt,H) is an irreducible unitary highest weight representation of G, 
is an irreducible (finite dimensional) unitary representation of K. The 
JC-modulc has an expression ct® XO) where a G K such that a\c is trivial 
and xo is a unitary character of K. 

Let x' be a unitary character of K such that x' is trivial on the center 
Zq of G (namely, x' is well-defined as a representation of Ada{K) ~ K/Zq). 
For later purposes, we take x' such that —\/^dx'{Z) 0. There exists an 
irreducible finite dimensional representation F of G such that ~ a (g) x' 



Multiplicity-free restrictions to symmetric pairs 



31 



as iiT-modules because cr %' is well-defined as an algebraic representation of 

We set X '■= Xo® ix')* of K. Because —\/^^dx{Z) <C 0, the irreducible 
highest weight {Qc,K)-modvle V such that (V^')''+ — x is unitarizable. Let 
(tt'jTY') denote the irreducible unitary representation of G whose underlying 
(gc, -?^) -module Ti'j^ is isomorphic to V . Since Ti'j^ is an irreducible {qc,K)- 
module, F is a (gc, -ft')-niodule of finite length. Furthermore, as is 
a highest weight module, so arc all subquoticnt modules of ® F. Then, 
T-Lk arises as a subquoticnt of H'^ ® F because the JsT-module occurs as 
a subrepresentation of (7Y^ ® FY+ in view of 

~ a xo - X <^ X') - {'HkY^ ® FP+ C {H'k ® F)^+ . 
Hence, we have shown Lemma 5.3. □ 

5.4 Uniform estimate of multiplicities for tensor products 

Let {tTjX) be a (gc, -f'^)-iiiodule of finite length. This means that tt admits a 
chain of submodules 

= YaCYiC ■■■ cYn (5.4.1) 

such that Yi/Yi-i is irreducible for i = 1, . . . , iV. The number N is indepen- 
dent of the choice of the chain (5.4.1), and we will write 

m(7r) :=N . 

That is, m(7r) is the number of irreducible (flC) -^^)-inodules (counted with 
multiplicity) occurring as subquotients in tt. Here is a uniform estimate of 
m(7r) under the operation of tensor products: 

Proposition 5.4.1. Let F be a finite dimensional representation of a real 
reductive connected Lie group G. Then there exists a constant C = C{F) such 
that 

for any irreducible {gc, K)-module w. 

Before entering the proof, we fix some terminologies: 

Definition 5.4.2. We write J^{qc, K) for the category of (gcj i4r)-modules of 
finite length. The Grothendieck group V{qc,K) of T{qc-,K) is the abelian 
group generated by (flC) -f'')-niodules of finite length, modulo the equivalence 
relations 

X ~ y-i-z 

whenever there is a short exact sequence 
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O^Y ^ X ^ Z ^0 

of {gc, K)-modvL\es. Then 

induces a group homomorphism of abelian groups: 

m:V{sc,K)^Z. 

The Grothendieck group V(0c, K) is isomorphic to the free abelian group 
having irreducible (gc, i^)-niodules as its set of finite generators. 

Suppose (tt, X) is a {qc, if)-module of finite length. Then, in the Grothendieck 
group V{gc, K), we have the relation 

X = 0m,(y)r, (5.4.2) 

Y 

where the sum is taken over irreducible (fic, -f^)-inodules. Then we have 

m(7r) = ^m,(y). (5.4.3) 

Y 

Suppose (tt', X') is also a (gc, -fi')-modules of finite length. We set 

[tt : tt'] := dimHom(g,,^)(0m,(y)y,0m,,(y)F) (5.4.4) 

Y Y 

= J2mAY)mn'{Y). (5.4.5) 

Y 

The definition (5.4.4) makes sense in a more general setting where one of Xor 
X' is not of finite length. To be more precise, we recall from [41, Definition 
1.1]: 

Definition 5.4.3. Let A{gc,K) be the category of (gc, -?^)-inodules {n,X) 
having the following properties: 

1) (-fsT-admissibility) dimHom/^ (r, tt) < oo for any r G if. 

2) (discretely decomposability, see [41, Definition 1.1]) X admits an increas- 
ing filtration 

= Fo C Fi C Fa C • • • 
of gc-modules such that Yi/Yi-i is of finite length and that X = \J^i Yi. 

Wc refer the reader to [41] for algebraic results on discretely decomposable 

(gc, -ftr)-modulcs such as: 

Lemma 5.4.4. Suppose X G A{gc,K). 

1) Any submodule or quotient of X is an object of A{qc,K). 

2) The tensor product X ® F is also an object of A{qc,K) for any finite 
dimensional {QciK)-module. 
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For X e A{qic,K), we can take the filtration {Fj} such that Yi/Yi_i is 
irreducible as a {qc, K)-mo6.vle for any i. Then, for any irreducible {qc,K)- 
module, 

#{i : Yi/Yi-i is isomorphic to Y} 
is finite and independent of the filtration, which we will denote by m-^iy). 

Definition 5.4.5. Suppose X e A{qc,K). We say the {Qc,K)-m.o<i\x\e X is 
multiplicity-free if 

rriTriY) < 1 for any irreducible (gc, -ft')-module Y. 

This concept coincides with Definition 1.1 \i X is the underlying {qc,K)- 
module of a unitary representation of G. The point of Definition 5.4.5 is that 
wc allow the case where X is not unitarizable. 

Generalizing (5.4.5), we set 

[7r:7r'] ■.= Y,m^{Y)m^,{Y) 

Y 

for 7r,7r' G A{qctK). Here arc immediate results from the definition: 
Lemma 5.4.6. Let 7r,7r' e A{qciK). 

1) [tt : tt'] < oo if at least one of i: and it' belongs to J^{qC) K). 

2) dimHom(g^,i^)(7r,7r') < [tt : tt']. 

3) [tt : tt'] = [tt' : tt]. 

4) m-j^iY) = [tt : y] ifY is an irreducible {gc, K)-module. 

5) [tt : tt'] < m(7r) if n' is multiplicity-free. 

Now, we return to Proposition 5.4.1. 

Proof of Proposition 5.4-1- We divide the proof into three steps: 
Step 1 (tt is a finite dimensional representation): We shall prove 

m{Tri»F)<dimF (5.4.6) 

for any finite dimensional representation tt of G. 

Let b = t + u be a Borel subalgebra of gc with u nilradical. We denote by 
(u, V) the j th cohomology group of the Lie algebra u with coefficients in 

a u-module V. Since the Lie algebra b is solvable, we can choose a b-stable 

filtration 

F = FkD Fk-i D • • • D Fo = {0} 

such that diuiFi/ Fi-i — 1. 

Let us show by induction on i that 

dimF°(u,7r® Fj) < i. (5.4.7) 
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This will imply m(7r O F) = dimF°(u, tt O F) < fc = dimF. 

The inequality (5.4.7) is trivial if i = 0. Suppose (5.4.7) holds for i — 1. 
The short exact sequence of b-modules 

^ TT O Fi_i ^n^Fi^n® {Fi/Fi_i) 

gives rise to a long exact sequence 

^ H°{U, TT ® Fi_i) ^ 7J"(U, TT ® Fi) ^ i?°(u, TT {Fi/Fi_i)) 

^ H\u,TT<»Fi_i)^ ... 

of t-modules. In particular, we have 

dimijO(u,7r®Fi) < dimi?°(u, tt® F^.i) + dimi?°(u, tt® (Fj/Fj.i)) . (5.4.8) 

Because Fi/Fj_i is trivial as a u-module, we have 

ifO(u,7r0 (Fi/Fi_i)) = H%u,7r) ® {Fi/Fi_,). (5.4.9) 

By definition if°(u, tt) is the space of highest weight vectors, and therefore 
the dimension of the right-hand side of (5.4.9) is one. Now, the inductive 
assumption combined with (5.4.8) implies dimff''(u, tt (g) Fj) < i, as desired. 
Step 2 (tt is a principal series representation): In this step, we consider the 
case where tt is a principal series representation. We note that tt may be 
reducible here. 

Let P = LN be a Levi decomposition of a minimal parabolic subgroup P 

of G, W an irreducible (finite dimensional) representation of L, and Indp(W'^) 
the underlying (gci -ftr)-module of a principal series representation induced 
from the representation lEl 1 of P = LN (without p-shift). Then, the socle 
filtration is unchanged so far as the parameter lies in the equisingular set, 
and thus, there are only finitely many possibilities of the socle filtration of 
Indp(W^) for irreducible representations W of L. We denote by m{G) the 
maximum of m(Indp(T4^)) for irreducible representations W of L. 

Let F be a finite dimensional representation of G. Then we have an iso- 
morphism of (flc, -f^)-modules 

Ind${W) OF- Ind^(VF (g) F) , 

where F is regarded as a P-module on the right-hand side. We take a P-stable 
filtration 

Wn:=W(3FD Wn-i D • • • D Wo = {0} 

such that each Wi/Wi-i is irreducible as a P-module. We notice that n < 
dimF by applying Step 1 to the F-module F\l. As Indp(W(8)F) is isomorphic 
to 0"^i Indp(Wi/Wi_i) in the Grothendieck group V{gc, K), we have shown 
that 

m(Indp(W) (g) F) < n m(G) < (dimF) m(G) 
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for any irreducible finite dimensional representation W of L. 

Step 3 (general case): By Cassclman's subrcprcscntation theorem (see [87, 
Chapter 3]), any irreducible {Qc,K)-module n is realized as a subrepresenta- 
tion of some induced representation Indp(Vr). Then 

■m{n (g) F) < m(7r (g) lnd${W)) < C 

by step 2. Thus, Proposition 5.4.1 is proved. □ 

5.5 Proof of Theorem D 

Now let us complete the proof of Theorem D. 

Let TT = TTi 772 be an irreducible unitary highest weight representation 
of G' := G X G. It follows from Lemma 5.3 that there exist an irreducible 
unitary highest weight representation tt' = Tr[M of scalar type and a finite 
dimensional representation F of G' such that -kk occurs as a subquotient of 
tt'k ® F. 

By using the notation (5.4.4), we set [Vi : F2] := [{Vi)k ■ {V2)k] for 
G-modules Vi and V2 of finite length. Then, for n G G, we have 

fn-Ki,T,Alj) = dimHomG(^,7r|diag(G)) 

< [m : 7r|diag(G)] 

< [/x: (7r'0i^)|diag(G)] 

= (-F*|diag(G)) : 7r'|diag(G)] 

<m(Ai ® (F*|diag(G))) (5.5.1) 

< C{F*) . 

Here the inequality (5.5.1) follows from Lemma 5.4.6 (5) because 7r'|diag(G) — 
'k'i®it2 is multiplicity-free (see Theorem D (2)). In the last inequality, C{F*) 
is the constant in Proposition 5.4.1. This completes the proof of Theorem D 
(1). □ 

Remark 5.5. The argument in Subsections 8.8 and 8.9 gives a different and 
more straightforward proof of Theorem D. 

6 Counter examples 

In this section, we analyze the assumptions in Theorems A and B by coun- 
terexamples, that is, how the conclusions fail if we relax the assumptions on 
the representation tt. 

Let (G, H) be a reductive symmetric pair corresponding to an involutive 

automorphism t of G, and tt an irreducible unitary representation of G. We 
shall see that the multiplicity of an irreducible summand occurring in the 
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restriction 'k\h can be: 

1) greater than one if tt is not of scalar type (but we still assume that tt 
is a highest weight module); 

2) finite but not uniformly bounded if tt is not a highest weight module 

(but wc still assume that 'k\h decomposes discretely); 

3) infinite if 7r|// contains continuous spectra. 

Although our concern in this paper is mainly with a non-compact subgroup 
-ff , we can construct such examples for (1) and (2) even for H = K {a. maximal 
compact subgroup modulo the center of G). 

Case (1) will be discussed in Subsection 6.1, (2) in Subsection 6.2, and (3) 
in Subsection 6.3, respectively. To construct an example for (3), we use those 
for (1) and (2). 

6.1 Failure of multiplicity- free property 

Let G = Sp{2,M.). Then, the maximal compact subgroup K is isomorphic to 
U{2). We take a compact Cartan subalgebra t. Let {/i, /2} be the standard 
basis of y/^t* such that /i(0,t) = {±/i ± /2, ±2/i, ±2/2}, and we fix a 
positive system A'^{t, t) := {/i — /2}. In what follows, we shall use the notation 
(Ai, A2) to denote the character Ai/i + A2/2 of I. 

Given (p, q) G T? with p> q,we denote by tt^^*^^! the irreducible representa- 
tion of U{2) with highest weight {p, q) ~ p,fi + q,f2 - Then dimTr^^^^j = p — q+l. 

The set of holomorphic discrete series representations of G is parametrized 
by A := (Ai, A2) e with Ai > A2 > 0. We set fi = (Mi,At2) := (Ai + l,A2 + 2) 
and denote by = tt^^^^^^-* the holomorphic discrete series representation 
of G characterized by 

Z(g)-infinitesimal character =(Ai,A2) (Harish-Chandra parameter), 
minimal K-type = 7r|^^^^||^^^ (Blattner parameter). 

We note that tt^ is of scalar type if and only if /^i = ^2- 

Wc know from Theorem B that multiplicities of i^T-typc t occurring in 
are uniformly bounded for fixed /z = {111,122)- Here is the formula: 

Example 6.1 (upper bound of K -multiplicities of holomorphic discrete series). 
sup dimHomi<-(r, tt^Ik) 

I : , I right side of (6.1.1) = 1 if and only if either of the following two cases 
holds: 

/ii /i2 (i.e. TT^ is of scalar type), (6.1.2) (a) 

Ml = + 1 (i-e. TT^ is of two dimensional minimal if -type). (6.1.2) (b) 



Atl - /X2 



(6.1.1) 
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Thus, the branching law of the restriction 'k'^\k is multiplicity- free if and only 
if //i = /i2 or /ii = /i2 + 1- The multipUcity-free property for /ii = /i2 (i-e. 
for TT^ of scalar type) follows from Theorem A. The multiplicity-free property 
for /ii = /i2 + 1 is outside of the scope of this paper, but can be explained in 
the general framework of the 'vector bundle version' of Theorem 2.2 (see [47, 
Theorem 2], [49]). 

Proof. It follows from the Blattncr formula for a holomorphic discrete series 
representation ([32], [78]) that the iC-type formula of tt^ is given by 



S{C^) 

e - 

a>b>0 
(a,6)eN^ 



C/(2) 
(2a,26) 



(6.1.3) 



where K = U{2) acts on ~ S'^(C^) as the symmetric tensor of the natural 
representation. We write nn{p,q) for the multiplicity of the K-type T^^pf-f 
occurrmg m -k"^ = i^^l^l^^), that is, 

n^,{p,q)-= dim Hom^ {-^fp^q) .'^^\k) ■ 
Then, applying the Clebsch-Gordan formula (1-7.1) (f) to (6.1.3), we obtain 

n^(p,g) = #{(a,6) e : (a, 6) satisfies a > 6 > 0, (6.1.4) and (6.1.5)}, 
where 



max(2a -|- /i2, 26 -|- /xi) < p < 2a -|- /xi . 



(6.1.4) 
(6.1.5) 



In particular, for fixed (/Ui, /U2) and {p, q), the integer b is determined by a from 
(6.1.4), whereas the integer a satisfies the inequalities p — Hi < 2a < p — 112- 
Therefore, 



n^i{p,q) < 



(P-/X2)- (p-Mi) 



1 



Hi- H2 + 2 



□ 



6.2 Failure of uniform boundedness 



We continue the setting of Subsection 6.1. Let B be a Borel subgroup of 

Gc ^ Sp{2, C). Then, there exist 4 closed orbits of Kc ^ GL{2, C) on the full 
flag variety Gc/B. (By the Matsuki duality, there exist 4 open orbits of G = 
Sp{2, M) on Gc / B. This observation will be used in the proof of Example 6.3.) 
By the Bcilinson- Bernstein correspondence, we sec that there arc 4 scries 
of discrete series representations of G. Among them, two are holomorphic 
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and anti-holomorphic discrete series representations, that is, tt^ and (n^)* 
(the contragredient representation) with notation as in Subsection 6.1. The 
other two series are non-holomorphic discrete series representations. Let us 
parametrize them. For A := (Ai,A2) G (Ai > — A2 > 0), we write Wx for 
the discrete series representation of G characterized by 

Z(f|)-infinitesimal character =(Ai,A2) (Harish-Chandra parameter), 

minimal ii'-type = ''^^x^]_i (Blattner parameter). 

Then, non-holomorphic discrete scries representations arc cither Wx or its 
contragredient representation W'^ for some A e with Ai > — A2 > 0. We 
define a 6'-stablc Borcl subalgcbra q = tc + u of = tc + Pc such that 

Z\(u n Pc, t) := {2/1, /i + /2, -2/2} , A{n n tc, t) {fi " ./2} • 

Then, the Harish-Chandra module {Wx)k is isomorphic to the cohomologi- 
cal parabolic induction T^\{C(^Xi,X2)) of degree 1 as (gc, -f'')-niodules with the 
notation and the normalization as in [86]. We set /xi := Ai + 1 and fi2 '■= A2. 

Example 6.2 (multiplicity of K -type of non-holom,orphic discrete series Wx)- 

Ui2) 
(P>9) 

mx{p, q) ■■= dimHom^CTT^p^^j, Wa|x) • 
Then, mx{p, 9) 7^ only if (p, q) G 1? satisfies 

P > Ml , V — q> — [i-i and p - g e 2Z + /xi + /i2 • (6.2.1) 

Then, 

mx{p,q) = 1 + min( 



We write mx{p,q) for the multiplicity of the if- type 7r[^'^| occurring in Wx, 
that is, 

U{2) 



2 (6.2.2) 

In particular, for each fixed A, the JC-multiplicity in Wx is not uniformly 
bounded, namely, 

sup dimHom/<-(r, W^lx) = sup mx{p, q) = 00 . 

r^K {p^q) satisfies (6.2.1) 

Proof. For p,q gIj, we write C(p^g) for the one dimensional representation of 
tc corresponding to the weight pf\ + gr/2 S ^. According to the tc-module 
isomorphism: 

U n Pc - C(2,o) ® C(i4) ® C(o,-2) , 

the symmetric algebra ^(unpc) is decomposed into irreducible representations 
of tc as 



5(unpc)^ 5'^(C(2,o))®5''(C(i,i))®5^(C(o,_2)) 

a.b.cGN 

- C(2a+6,6-2c) • (6.2.3) 



a,b,c6P 
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We denote by H^u fl ^ctt) the jth cohomology group of the Lie algebra 

u n with coefficients in the u n fc-module tt. If tt is a 6c-modulc, then 
H^{u n 6c) tt) becomes naturally a tc-module. Then, Kostant's version of the 
Borel-Weil-Bott theorem (e.g. [85, Chapter 3]) shows that 

C(p,g) (j = 0) , 

C(,_i,p+i) 0- = l), (6.2.4) 

{0} (jV0,i). 

By using the Blattner formula due to Hecht-Schmid (e.g. [85, Theorem 6.3.12]), 
the K-type formula of W\ is given by 

mx{p,q) = dimRoniKin'^p^y Wx\k) 
1 

= ^(-l)^^ dimHomt,(if^ (u n «c, Trgg), S{u n pc) ® C(^„^,)) . 

3=0 

Now, comparing (6.2.3) with the above formula (6.2.4) as tc-modules, we see 

mx{p, q) = #{(a, b,c) eN^ : p = 2a + b + iJ,i,q = b - 2c + (12} 

- #{{a,b,c) eN^ :q-l = 2a + b + fiup+l = b-2c + fi2} 

= #{(0, b,c) :p = 2a + b + iJ.i,q = b-2c + fi2} 

p-q- 111+ H2 . 
2 

Thus, the formula (6.2.2) has been verified. □ 
6.3 Failure of finiteness of multiplicities 

Multiplicities of the branching laws can be infinite in general even for reductive 
symmetric pairs (G, H). In this subsection, we review from [43, Example 5.5] 
a curious example of the branching law, in which the multiplicity of a discrc;te 
summand is non-zero and finite and that of another discrete summand is 
infinite. Such a phenomenon happens only when continuous spectra appear. 

Example 6.3 (infinite and finite multiplicities). Let (Gc,G) be a reductive 
symmetric pair (5^(2, C), S'p(2, M)). We note that (Gc,G) is locally isomor- 
phic to the symmetric pair (50(5, C), 50(3, 2)). We take a Cartan subgroup 
H = TAoiGc. We note that T ~ and A ~ and identify f with I?. 

Let w = ^® unitary principal series representation of Gc 

induced unitarily from the character x of a Borel subgroup B containing 
H = TA such that 



^'■("nec,<,2) = < 



= 1 -|- min( 



p- IXi 
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We assume a,b>0 and set 

c(aH) M2; a> b) := #{{s,t,u) £ : a = + 2s + t, b = 112 + t + 2m} . 

Then, the discrete part of the branching law of the restriction ^^f,)'"''* |sp(2,M) 
is given by the following spectra: 

c{^il,^^2;aM^^^iiy{^Zif^^^ ^iw,®w^,), (6.3.1) 

Mi>M2>3 Ai>-A2>0 

with the notation as in Examples 6.1 and 6.2. 

The first term of (6.3.1) is a finite sum because there are at most finitely 
many (/xi,/X2) such that c{iJ,i, iJ,2',a,b) ^ for each fixed {a,b). For instance, 
the first term of (6.3.1) amounts to 

-(^af® {-S^TT (-ultiphcity-free) 

S<fj.i<a 3<)Lti<a 
fx\=a mod 2 fx\=a mod 2 

if 6 = 3. 

The second term of (6.3.1) is nothing other than the direct sum of all 
non-holomorphic discrete series representations of G = 5^(2, M) with infinite 
multiplicities for any a and b. 

Sketch of Proof. There exist 4 open G-orbits on Gc/B, for which the isotropy 
subgroups are all isomorphic to T ~ T^. By the Mackey theory, the restriction 
'^fah)\G is unitarily equivalent to the direct sum of the regular representations 
realized on L^-sections of G-equivariant line bundles G Xt C(_±a,±b) ~^ G/T. 
That is, 

<a%lG^ L^(G/T,C(,,„,e.6)). 

ei,e2=±l 

Therefore, an irreducible unitary representation u of G occurs as a dis- 
crete spectrum in '^^^b)\G if ^^id only if a occurs as a discrete summand 
in L'^{G/T, C(£j£, e^;,)) for some £i, £2 = ±1- Further, the multiplicity is given 

by 

dimHomG(o-,ro^';^^|G) = E dimHomT2(C(e^o,e26), ctIts) 

£l,£2=±l 

by the Frobenius reciprocity theorem. 

Since T is compact, a must be a discrete series representation of G = 
5*^(2, R) if fj occurs in L?{G/T, C(eja,e2&)) ^ ^ discrete summand. We divide 
the computation of multiplicities into the following two cases: 

Case I. cr is a holomorphic series representation or its contragredient rep- 
resentation. Let a = 77^^^^'"*^ Combining (6.1.3) with the weight formulae 
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5(C3)|t2~ 5^(C(2,o))0^*(C(i,i))^5"(C(o,2))- C(2,+t,t+2„) , 
p+g=/ii+/i2 

A'2<P<(Iil 

we have 

dim HoniT2 (C(„,b) , tt^^ ^^.R) ^ ^ ^^^^ , M2 ; a, 6) • 

Case II. cr is a non-holomorphic discrete series representation. Let a = Wx- 
It follows from the JsT-type formula (6.2.2) of W\ that we have 

dimHomT2(C(a,6),iyA|T2) = ^ TOaCp, g) dimHomT2 (C(a,6), 71^^^^]) = oo . 

Likewise for a = (the contragredient representation). 

Hence, the discrete part of the branching law is given by (6.3.1). □ 



7 Finite Dimensional Cases — Proof of Theorems E and 
F 

7.1 Infinite v.s. finite dimensional representations 

Our method applied to infinite dimensional representations in Sections 3 and 
4 also applies to finite dimensional representations, leading us to multiplicity- 
free theorems, as stated in Theorems E and F in Section 1, for the restriction 
with respect to symmetric pairs. 

The comparison with multiplicity-free theorems in the infinite dimensional 
case is illustrated by the following correspondence: 

a non-compaet simple group G -(-^ a compact simple group Gu 

a unitary highest weight module <-> a finite dimensional module 

scalar type (Definition 1.3) ^ "pan type" (Definition 7.3.3) 

Theorems A and B Theorems E and F. 

The main goal of this section is to give a proof of Theorems E and F by 
using Theorem 2.2. Geometrically, our proof is built on the fact that the Hu 
action on the Hermitian symmetric space is strongly visible if {Gu,Hu) is a 
symmetric pair (see [50]). 



7.2 Representations associated to maximal parabolic subalgebras 

Let 0c be a complex simple Lie algebra. We take a Cartan subalgebra j of 
Qc, and fix a positive system Z\+(gcj ))• We denote by {ai, .... q„} the set of 
simple roots, and by {wi, . . . , w„} (c )*) the set of the fundamental weights. 
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We denote by irreducible finite dimensional representation of flc with 
highest weight A = '^^^^miUJi for mi,...,m„ G N. It is also regarded as 
a holomorphic representation of Gc, a simply connected complex Lie group 
with Lie algebra flc- 

We fix a simple root ai, and define a maximal parabolic subalgebra 

PiC ■= Uc + nr^ 

such that the nilradical n^^ and the Levi part [jc ( 3 j) are given by 

A{Uc,j) = Z-span of {ai, . . . ,Q!„} n A{qc,)) , 
A{n-c,j) = A-{gc,j)\A{kc,j). 

We shall see that irreducible finite dimensional representations realized on 
generalized flag varieties Gc/Pc is multiplicity- free with respect to any sym- 
metric pairs if Pc has an abelian unipotent radical. 

Wc write P-^ = LicN^ for the corresponding maximal parabolic subgroup 
of Gc. 

Let Hom(p^,C) be the set of Lie algebra homomorphisms over C. Since 
any such homomorphism vanishes on the derived ideal [p^,p^], Hom(pT^,C) 
is naturally identified with 

Hom(pr^/[p-c,prJ,C) ~ Cui . 

Next, let Hom(P^,C^) be the set of complex Lie group homomor- 
phisms. Then, we can regard Hom(P.^,C^) C Hom(p^,C). As its subset, 
Hom(P.p,C^) is identified with Zcji since Gc is simply connected. 

For fc e Z, we write Cku>i for the corresponding character of P^, and 
denote by 

^kui ■= Gc X p- 'Ckui Gc/P^ (7.2.1) 

the associated holomorphic line bundle. We naturally have a representation 
of Gc on the space of holomorphic sections O{£koji)- Then, by the Borel- 
Weil theory, 0{Cku>^) is non-zero and irreducible if fc > and we have an 
isomorphism of representations of Gc (also of gc): 

<l,^0{Lk.,). (7.2.2) 

7.3 Parabolic subalgebra with abelian nilradical 

A parabolic subalgebra with abelian nilradical is automatically a maximal 
parabolic subalgebra. Conversely, the nilradical of a maximal parabolic subal- 
gebra is not necessarily abelian. We recall from Richardson-Rohrle-Steinberg 
[75] the following equivalent characterization of such parabolic algebras: 
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Lemma 7.3.1. Retain the setting of Subsection 7.2. Then, the following four 

conditions on the pair (flCjQfj) are equivalent: 

i) The nilradical is abelian. 

ii) {qc, kc) is a symmetric pair. 

iii) The simple root ai occurs in the highest root with coefficient one. 

iv) (flcaj) is in the following list if we label the simple roots ai,. .. ,an in 
the Dynkin diagram as in Table 7.3.2. 

(7.3.1) Type An ai, a2, . . . , q;„ 

(7.3.2) Type B„ ai 

(7.3.3) Type Cn oin 

(7.3.4) Type Dn ai,a„_i,a„ 

(7.3.5) Type Eq ai,Q!6 

(7.3.6) Type E-j a-j 

For types G2, Fi, Eg, there are no maximal parabolic subalgebras with abelian 
nilradicals. 



Table 7.3.2. 

(An) 
(Bn) 
(Cn) 



ai a2 a„-i a„ 



ai a2 CXn-l Oin 



Oil Oi2 



(Du) 



{Ee) 



(Er) 




ai a2 an-2 a. 



012 



Oi\ Oiz 04 015 ae 
) a2 



ai as Q4 Q5 CX6 CX7 



Proof. Sec [75] for the equivalenee (i) <^ (iii) ^ (iv). The imphcation (iv) ^ 
(ii) is straightforward. For the convenience of the reader, we present a table 
of the symmetric pairs {qc, kc) corresponding to the index i in (iv). 
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Type 0c Uc i 

An 5[(n+lX) s[(i,C)+sl(n + l-i,C)+C i = l,2,...,n 

Bn so(2n + l,C) so(2n-l,C) + C i=l 

C„ sp(n,C) gl{n,C) i = n 

Dn so(2n,C) so(2n-2,C) + C i = l 

so(2n, C) 0[(n, C) 2 = ^—1,^1 

Ed ee 5o(10,C) + C z = l,6 

i?7 e? ee + C i = 1 

If (flc, Uc) is a symmetric pair, then [n^^,nj^] C n^:^ n Uc = {0}, whence (ii) 

Definition 7.3.3. We say the representation irf^, (fc = 0, 1, 2, . . . ) is of pan 
type, or a pan representation if [qctOh) satisfies one of (therefore, all of) the 
equivalent conditions of Lemma 7.3.1. Here, pan stands for a parabolic sub- 
algebra with abelian nilradical. 

7.4 ExEimples of pan representations 

Example 7.4- Let gc = 0[("-: and A = (Ai, . . . , A„) G Z" with Ai > A2 > 
• • • > A„. (This 0c is not a simple Lie algebra, but the above concept is defined 
similarly.) Then, tta is of pan type if and only if 

Ai = • • • = Ai > Aj+i = • • • = A„ 

for some i (1 < z < n — 1). Then, (l)jc — C) + 0l(n — z, C). 

In particular, the fcth symmetric tensor representations 5'^(C") (fc e N) 

and the fcth exterior representations /l'^(C") (0 < A: < n) arc examples of 
pan representations since their highest weights are given by (A;, 0, ... , 0) and 
(!,...,!, 0, . . . , 0), respectively. 

k n—k 

S. Okada [68] studied branching laws for a specific class of irreducible finite 
dimensional representations of classical Lie algebras, which he referred to as 
"rectangular-shaped representations" . The notion of "pan representations" is 
equivalent to that of rectangular-shaped representations for type {An), {Bn), 
and (C„). For type (Dn), 7rfcw„_i, TTfct^^ (A: G N) are rectangular-shaped repre- 
sentations, while TTfeoji {k € N) are not. 



7.5 Reduction to rank condition 



Suppose {gc,ai) satisfies the equivalent conditions in Lemma 7.3.1. Let 9 be 
the complex involutive automorphism of the Lie algebra 0c that defines the 
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symmetric pair {qc, [jc)- We use the same letter to denote the corresponding 
holomorphic involution of a simply connected Gc- We take a maximal compact 
subgroup Gu of Gc such that 0Gu = Gu- Then K := G^ = Gc/flLjc becomes 
a maximal compact subgroup of Lic- 

Let r be another complex involutivc automorphism of gc, and [gc, f)c) the 
symmetric pair defined by r. We also use the same letter r to denote its lift 
to Gc- We recall from Subsection 3.7 the 'twisted' involution for g e Gc is 
given by 

T^ix) = gT{g-^xg)g-^ (x G Gc) • 
Lemma 7.5. Let {9, r) be as above. 

1) There exist an involutive automorphism a of Gu and g G Gc satisfying 
the following three conditions (by an abuse of notation, we write r for t^): 

(7.5.1) TQu = Qu, crQ = Oa, ar = ra. 

(7.5.2) The induced action of a on Gu/K is anti-holomorphic. 

(7.5.3) (fl!/)'^'^^'"^ contains a maximal abelian subspace in {5u)~'^'~^ ■ 

2) For any x G Gu/K, there exists h G (G^)o such that a{x) = h ■ x. In 
particular, each {G'[j)o-orbit on Gu/K is preserved by a. 

Proof. 1) See [50, Lemma 4.1] for the proof. 

2) The second statement follows from the first statement and a similar argu- 
ment of Lemma 3.3. 

7.6 Proof of Theorem E 

We are now ready to complete the proof of Theorem E in Section 1. 

Let TT = TT^I^. be a representation of pan type. As in Subsection 7.2, we 
consider the holomorphic line bundle Cku^ Gc/Pj£ and realize tt on the 
space of holomorphic sections 0{£ku,i)- We fix a G^-invariant inner product 
on 0{£kuj-). With notation as in Subsection 7.5, we have a diffeomorphism 

Gu/K::,Gc/P,^, 

through which the holomorphic line bundle Ckui ~> C^c/P/c naturally iden- 
tified with the G(7-equivariant holomorphic line bundle C ^ D, where we set 
C := Gu Xk Cfeo;; and D := Gu/K (a compact Hermitian symmetric space). 

Now, applying Lemma 7.5, we take a and set H := (G^)o. We note that the 
complexification of the Lie algebra of H is equal to ()c up to a conjugation by 
Gc- By Lemma 7.5, the condition (2.2.3) in Theorem 2.2 is satisfied. Further- 
more, we see the condition (2.2.2) holds by a similar argument of Lemma 9.4. 
Therefore, the restriction t^\(Gu)1 ^® multiplicity-free by Theorem 2.2. Hence, 
Theorem E holds by Weyl's unitary trick. □ 
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7.7 Proof of Theorem F 

Suppose TTi and TT2 are representations of pan type. We realize tti and TT2 on 
the space of holomorphic sections of holomorphic hne bundles over compact 
symmetric spaces Gu/Ki and Gu/K2 - respectively. We write 9i for the corre- 
sponding involutive automorphisms of Gu that define Ki (i = 1, 2). In light of 
Lemma 7.3.1 (iv), we can assume that 6162 = ^2^1- Then, applying Lemma 7.5 
to {61,62) we find an involution a' G Aut(G!7) satisfying the following three 
conditions: 

(7.7.1) a'ei = eia' (1 = 1,2). 

(7.7.2) The induced action of a' on Gjj/Ki (i = 1,2) is anti-holomorphic. 

(7.7.3) {qu)'^ -Si -02 contains a maximal abelian subspace of {qu)~^^'~^^ ■ 
We remark that the condition (7.7.2) for z = 2 is not included in Lemma 7.5, 
but follows automatically by our choice of a. 

We define three involutive automorphisms r, and a on Gu x Gjj by 
t{9i,9i) ■= {91, gi), := {61,62) and a := {a',u'), respectively. Then {Gu x 
Gu)'^ = diag((j!7). By using the identification 

{Qu ® flc/)-" = {{X, -X):Xe Qu] ^ 0u , {X, -X)^X, 
we have isomorphisms 

{qu ® 9u)-^'-' - {9u)-'"-'' , 
{3u®&u)^'-^'-'^{Su)''' 

Thus, the condition (7.7.3) implies that {gu ® Qu)'^'~'^'~^ contains a maximal 
abelian subspace of {gu © 0c/)~^'~^- Then, by Lemma 7.5 and by a similar 
argument of Lemma 3.3 again, for any {x,y) G Gu/Ki x Gu/K2 there exists 
B, g G Gu such that cr'{x) = g ■ x and cr'{y) = g ■ y simultaneously. Now, 
Theorem F follows readily from Theorem 2.2. □ 

7.8 List of multiplicity-free restrictions 

For the convenience of the reader, we present the list of the triple {gc, i)c,i) 
for which wc can conclude from Theorem E that the irreducible finite dimen- 
sional representation tt^^^ of a simple Lie algebra gc is multiplicity-free when 
restricted to for any A; € N by Theorem E. 
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0c 


fic 


i 


sl(n + 1,C) 


s[(p, C) + si{n + 1 - p, C) + C 


1,2,..., n 


sl(n + 1,C) 


so(n + l,C) 


1,2,. ..,n 


5l(2m,C) 


sp(m, C) 


1,2, ...,2m- 1 


so(2n-|-l,C) 


so{p, C) + so(2n + 1 - p, C) 


1 


sp(n,C) 


sp(p,C)+sp(n-p,C) 


n 


sp(n,C) 


5[(n, C) 


n 


so(2n,C) 


5o(p,C) +so(2n -p, C) 


l,n — l,n 


so(2n,C) 


gl{n,C) 


1,11—1,71 


Ce 


so(10,C) +so(2,C) 


1,6 


ee 


sl(6,C) +sl(2,C) 


1,6 


Ce 


U 


1,6 


ee 


5p(4,C) 


1,6 


C7 


ee +so(2,C) 


7 




so(12,C) +sl(2,C) 


7 


t7 


Bl(8,C) 


7 



Some of the above cases were previously known to be multiplicity-free by 

casc-by-case argument, in particular, for the case rankgc = rankf)c- Among 
them, the corresponding explicit branching laws have been studied by S. 
Okada [68] and H. Alikawa [1]. 

There are some few representations tt that are not of pan type, but are 
multiplicity-free when restricted to symmetric subgroups H. Our method still 
works to capture such cases, but we do not go into details here (see [46, 51, 52]). 

8 Generalization of the Hua— Kostant— Schmid Formula 

This section discusses an explicit irreducible decomposition formula of the 
restriction -k\h where the triple (tt, G, H) satisfies the following two conditions: 

1) TT is a holomorphic discrete series representation of scalar type (Defini- 
tion 1.3). 

2) (G, H) is a symmetric pair defined by an involution r of holomorphic 
type (Definition 1.4). 

We know a priori from Theorem B (1) that the branching law is discrete 
and multiplicity-free. The main result of this section is Theorem 8.3, which 
enriches this abstract property with an explicit multiplicity-frcc formula. The 
formula for the special case H = K corresponds to the Hua-Kostant-Schmid 
formula ([26, 32, 78]). We also present explicit formulas for the irreducible 
decomposition of the tensor product representation (Theorem 8.4) and of the 
restriction U{p,q) i U{p — l,q) (Theorem 8.11). 
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Let us give a few comments on our proof of Theorem 8.3. Algebraically, 

our key machinery is Lemma 8.7 which assures that the irreducible G- 
decomposition is determined only by its JsT-structure. Geometrically, a well- 
known method of taking normal derivatives (e.g. S. Martens [63], Jakobsen- 
Vcrgnc [31]) gives a general algorithm to obtain branching laws for highest 
weight modules. This algorithm yields explicit formulae by using the obser- 
vation that the fiber of the normal bundle for jK^ C GjK is the tangent 
space of another Hermitian symmetric space C^^ / K'^ . The key ingredient of 
the geometry here is the following nice properties of the two symmetric pairs 
(G,G^) and (G,G^^): 

&) Kr^G^ ^KnG'''^, 
b) P = (pnfl^)©(pnfl^^). 

Unless otherwise mentioned, we shall assume H is connected, that is, H = 
Gq throughout this section. 

8.1 Notation for highest weight modules 

We set up the notation and give a parametrization of irreducible highest 
weight modules for both finite and infinite dimensional cases. 

First, we consider finite dimensional representations. Let us take a Cartan 
subalgebra t of a reductive Lie algebra t and fix a positive system Z\+ (6, 1) . 
We denote by tt* the irreducible finite dimensional representation of I with 
highest weight y^, if /i is a dominant integral weight. A t-module tt^ will be 
written also as 7r5 if the action lifts to K. 

Next, let G be a connected reductive Lie group, 9 a Cartan involution, 
K = {g & G : 6g = g} , Q = i + p the corresponding Cartan decomposition 
and 0c = 4c + Pc its complexification. We assume that there exists a central 
element Z oil such that 

0c = «c + P+ + P- (8.1.1) 

is the eigenspace decomposition of -^7= ad(Z') with eigenvalues 0, 1, and —1, 
respectively. This assumption is satisfied if and only if G is locally isomorphic 
to a direct product of connected compact Lie groups and non-compact Lie 
groups of Hermitian type (if G is compact, we can simply take Z = 0). 
We set 




(8.1.2) 



As in Definition 1.3, we say an irreducible (gc, .f^)-inodule F is a highest 
weight module if 

= {v eV -.Yv = for all Y e p+} 

is non-zero. Then, y+ is irreducible as a JsT-module, and the {gc, K)-modu\e 
V is determined uniquely by the ii'-structure on y+. If /i is the highest 
weight of yP+, we write V as tt^. That is, the irreducible (flC) .f^)-inodule tt^ 
is characterized by the iiT-isomorphism: 
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(7rfl)P+ :^ nl . (8.1.3) 

An irreducible unitary liighcst weiglit representation n oi G will be denoted 
by TT^ if the underlying (gci ii')-module of tt is isomorphic to tt^. Let Aq be 
the totality of /i siicli that tt^^ lifts to an irreducible unitary representation of 
G. For simply connected G, irreducible unitary highest weight representations 
were classified, that is, the set Aq (c \/— It*) was explicitly found in [12] and 
[30] (see also [13]). In particular, we recall from [12] that 

X{Z) e M for any X G Ag 

and ^ 

CG := sup X{Z) < oo (8.1.4) 

if G is semisimple. 

The highest weight module tt^ is the unique quotient of the generalized 
Verma module 

7V«(M):=f/(0c)®[/(Je+P+)<, (8.1.5) 

where tt' is regarded as a module of the maximal parabolic subalgebra 6c + 
p-i- by making p+ act trivially. Furthermore, tt^^ has a Z(0c)-iiifinitesimal 
character jj, + pg G via the Harish- Chandra isomorphism 

Homc-algcbra(2'(0c),C) ~ tc/W , 

where Z{qc) is the center of the enveloping algebra U{gc), W is the Weyl 
group of the root system A{Q,t), and pg is half the sum of positive roots 
A+{g,t):= A+{t,t)uA{p+,t). 

8.2 Strongly orthogonal roots 

Let G be a non-compact simple Lie group of Hermitian type, and t an invo- 
lution of holomorphic type which commutes with the Cartan involution 9. 
We take a Cartan subalgebra of the reductive Lie algebra 

r ■.= {X et-.rX = X} 

and extend it to a Cartan subalgebra t of t. We note that = l'^ fl t. The pair 
(6, F) forms a reductive symmetric pair, and t plays an analogous role to the 
fundamental Cartan subalgebra with respect to this symmetric pair. Thus, 
using the same argument as in [84], we see that if a e A{t, t) satisfies a\iT = 
then a = 0. Thus, we can take positive systems Zi+(l, t) and A'^{t^,V) in a 
compatible way such that 

a\t.&A+{r,e) iiaeA+{t,i). (8.2.1) 

Since r is of holomorphic type, we have tZ = Z, and therefore rp^ = p+. 
Hence, we have a direct sum decomposition p+ = p!j_ © p^"^, where we set 
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:= {X G p+ : rX = ±X} . 

Let us consider the reductive subalgcbra g'^^ . Its Cartan decomposition is 
given by 

= {Q-f> n fl^) + (0-^ n 0-^) = r + p-^ , 

and its complexification is given by 

Sc=^c®P7®P-"- (8-2-2) 

The Cartan subalgebra V of F is also a Cartan subalgebra of g^^. 

Let Zi(p^'^,t'^) be the set of weights of p^^ with respect to t"^. The roots 
a and (3 are said to be strongly orthogonal if neither a + (3 nor a — /3 is a 
root. We take a maximal set of strongly orthogonal roots {z/i, 1/2, . . . , i^;} in 
^(p+'^,f^) such that 

i) Ui is the lowest root among the elements in Zi(p^'^, f^), 

ii) i/j+i is the lowest root among the elements in Z\(p^'^,t^) that are 
strongly orthogonal to vi^ . . . ^Vj. 

A special case applied to r = ^ shows F = J, = t, p""^ = p, and 
A{p~^'^ , t^) = ^(p+, t). In this case, we shall use the notation {i/i, P2, • • • , f'rl 
for a maximal set of strongly orthogonal roots in zi(p+, t) such that 

i) vi is the lowest root among Z\(p4., t), 

ii) is the lowest root among the elements in Z\(p+,l) that are strongly 
orthogonal to i^i, . . . , i^j (1 < J < /)• 

Then, Z = R- rankg by [57]. Likewise, in light of (8.2.2) for the Hermitian sym- 
metric space C^/C^ r\K = G^^/G^'^, we have I = M-rankg^^. In general, 
I < I. 

8.3 Branching laws for semisimple symmetric pairs 

It follows from (8.1.3) that the highest weight module tt^ is of scalar type, 
namely, (77^)''+ is one dimensional, if and only if 

(At, a) = for any a G A{t, t) . (8.3.1) 

Furthermore, the representation is a (relative) holomorphic discrete series 
representation of G if and only if 

{fi + Pg,a)<0 for any a e Z\(p+,t) . (8.3.2) 

We are now ready to state the branching law of holomorphic discrete series 
representations ir^ of scalar type with respect to semisimple symmetric pairs 
{G,H): 
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Theorem 8.3. Let G be a non- compact simple Lie group of Hermitian type. 

AssuTiic that /i e \/— 1 1* satisfies (8.3.1) and (8.3.2). Let t he an involutive 
automorphism of G of holomorphic type, H = Gq ( the identity component of 
G'^), and {vi, . . . ,1/1} be the set of strongly orthogonal roots in Z\(p^'^, V) as 
in Subsection 8.2. Then, decomposes discretely into a multiplicity-free sum 
of irreducible H -modules: 

tt'^\h — ""^j^.j-;! a u- (discrctc Hubert sum). (8.3.3) 

oi>"->ai>0 
oi,...,aieN 

The formula for the case H = K (that is, t = 9) was previously found by 
L.-K. Hua (implicit in the classical case), B. Kostant (unpublished) and W. 
Schmid [78] (see also Johnson [32] for an algebraic proof). In this case, each 
summand in the right side is finite dimensional. 

For T 9, some special cases have been also studied by H. Jakobsen, M. 
Vergne, J. Xie, W. Bertram and J. Hilgert [7, 30, 31, 89]. Further, quantitative 
results by means of reproducing kernels are obtained in [5]. The formula (8.3.3) 
in the above generality was first given by the author [39] . 

We shall give a proof of Theorem 8.3 in Subsection 8.8. 



8.4 Irreducible decomposition of tensor products 

As we saw in Example 3.2.1, the pair (G x G, diag(G')) forms a symmetric 
pair. Correspondingly, the tensor product representation can be regarded as 
a special (and easy) case of restrictions of representations with respect to 
symmetric pairs. This subsection provides a decomposition formula of the 
tensor product of two holomorphic discrete series representations of scalar 
type. This is regarded as a counterpart of Theorem 8.3 for tensor product 
representations. 

We recall from Subsection 8.2 that {Di, . . . , i/^} is a maximal set of strongly 
orthogonal roots in Z\(p+,t) and 1 = M-rankfl. 

Theorem 8.4. Let G be a non-compact .simple Lie group of Hermitian type. 
Assume that /ii,/Lt2 G V— 1 1* satisfy the conditions (8.3.1) and (8.3.2). 
Then, the tensor product representation tt^^ <8i7r^2 decomposes discretely into 
a multiplicity-free sum of irreducible G-modules: 



ai>"'>ai>0 
ai,...,af e N 



The proof of Theorem 8.4 will be given in Subsection 8.9. 
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8.5 Eigenvalues of the central element Z 

Our proof of Theorems 8.3 and 8.4 depends on the algebraic lemma that the 
/C-type formula determines the irreducible decomposition of the whole group 
(sec Lemma 8.7). This is a very strong assertion, which fails in general for non- 
highest weight modules. This subsection collects some nice properties peculiar 
to highest weight modules that will be used in the proof of Lemma 8.7. 
For a ii'-module V , we define a subset of C by 

Spec^(y) := {eigenvalues of ZonV}, 

where we set 

For instance, Spec^{V) is a singleton if V is an irreducible ii'-module. We 
also note that Specg(£lc) = {0, ±1} by (8.1.1). 

Lemma 8.5. Suppose V is an irreducible {gc,K) -module. Then, 

1) Spcc^(V) C ao + Z for some ao € C. 

2) // sup Re Spec2(^) < oo, then V is a highest weight module. 

3) If V is a highest weight module n^, then Spec^(^) C — N + X{Z) and 
supReSpec^(F) = ReX{Z). 

4) IfV is a unitary highest weight module, then Spec^(V) C (— oo,cg], where 
cg is a constant depending on G. 

5) // both V and F are highest weight modules of finite length, then any irre- 
ducible subquotient W ofV®F is also a highest weight module. 

Proof. 1) For a S C, wc write the cigcnspace oi Z BsVa ■= {v & V : Zv = av}. 
Then, it follows from the Leibniz rule that 

P + Va C Va+l , tcVa C Va , and p-Va C K-1 • 

An iteration of this argument shows that 

Spcc^(C/(0c)K) C a + Z. 

Now we take ao such that Vag ^ {0}. Since V is irreducible, wc have V = 
U{gc)Vao^ and therefore Spec^(F) C oq + Z. 

2) Suppose supReSpcc^(l/) < oo. Since RcSpcc2(V^) is discrete by (1), there 
exists a € Spec^(y) such that Re a attains its maximum. Then 

P+Va C Va+l = {0} . 

Thus, Va C y+. Hence, V is a highest weight module. 

3) The highest weight module tt^ is isomorphic to the unique irreducible quo- 
tient of the generalized Verma module N^{X) = U{gc) 'Siu{{c+p+) 
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Poincare-Birkhoff-Witt theorem, N^{X) is isomorphic to 5'(p_) (S) tt' as a 
t-module. Thus, any t-type tt* occurring in tt^ is of the form 

fi = X+ ^ rUaa 

aeA{p_,t) 

for some nia S N. As a{Z) = — 1 for any a G Zi(p_,t), we have 

liiZ) = X{Z)- nia. (8.5.1) 

In particular, we have the following equivalence: 

Re/i(Z) = ReA(Z) ^ fj, = X, (8.5.2) 

and we also have 

Spec^(7r^) C {A(Z) - ^ nia : nia e M} = -N + X^Z) . (8.5.3) 

Furthermore, since the t-typc tt* occurs in nf, we have X{Z) e Spcc2(7r^). 
Hero, supReSpec^(7r^) = ReA(Z). 

4) This statement follows from (8.1.4) and from (3). 

5) For two subsets A and B in C, we write A+B := {a + b £ C : a e A,b e B}. 
Then, Spec^(y O F) c Specg(y) + Spec^{F) . Therefore, 

sup Re Specg (W) < sup Re Specg {V) 

< supReSpecg(F) + supReSpec^(i^) < oo. 

Hence, W is also a highest weight module by (2). □ 
8.6 Bottom layer map 

The following lemma finds an irreducible summand ('bottom layer') from the 

if-typc structure. 

Lemma 8.6. Let V be a (gc) K)-module. We assume that V decomposes into 
an algebraic direct sum of (possibly, infinitely many) irreducible highest weight 
modules. We set 

Supp((y) := e x/=Tr : Homt(7r* , V) ^ {0}} . 

If the evaluation map 

Suppt (y) ^ M , /i Re /i(Z) 
attains its maximum at /xq, then 

Hom(,„K)(7r8^,l/)^{0}. 
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Proof. Take a non-zero map q G Hom{(7r^^, V). As V is an algebraic direct 

sum of irreducible highest weight modules, there exists a projection p : V ^ 
77^ for some A such that p o q ^ 0. This means that tt^^^ occurs in tt^, and 
therefore we have 

Re^io{Z) < supRcSpcc^(7r^) = ReA(Z) . 

Here, the last equality is by Lemma 8.5 (3). 

Conversely, the maximality of no implies that Re/xo(-^) > ReX{Z). Hence, 
Re^o{Z^) = Re\{Z), and we have then /zg = A by (8.5.2). Since tt^ is an 
irreducible summand of V, we have Hom^g^. x)(7r^o, ^) {0}- ^ 

8.7 Determination of the flc-structure by X-types 

In general, the i^-type formula is not sufficient to determine the irreducible 
decomposition of a unitary representation even in the discretely decomposable 
case. However, this is the case if any irreducible summand is a highest weight 
module. Here is the statement that we shall use as a main machinery of the 
proof of Theorems 8.3 and 8.4. 

Lemma 8.7. Suppose {n.'H) is a K -admissible unitary representation of G, 
which splits discretely into a Hilbert direct sum, of irreducible unitary highest 
weight representations of G. Let Hk be the space of K -finite vectors ofH. 
Assume that there exists a function : — > N such that Hk is isomorphic 
to the following direct sum as t-modules: 

Hk — ^^nTr{X)TT^ (algebraic direct sum). (8.7.1) 

A 

Then, ^^(A) ^ only if X € Aq, that is, tt^ lifts to an irreducible unitary 
representation tt^ of G. Furthermore, the identity (8.7.1) holds as a {qc,K)- 

module isomorphism,, and, the unitary representation tt has the following de- 
composition into irreducible unitary representations of G: 

TT ~ n7r(A)7r^ (discrete Hilbert sum). (8.7.2) 

A 

Proof. We write an abstract irreducible decomposition of H as 

H ^ iTT-xTTx (discrete Hilbert sum). 

xeAc 

Since H is i^-admissible, the multiplicity mx < oo for all A, and we have 
an isomorphism of (gci -f^)-inodules with the same multiplicity m\ (see [43, 
Theorem 2.7]): 

Hk — mxTT^ (algebraic direct sum). (8.7.3) 
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Let us show n.^{X) = m\ for all A. For this, we begin with an observation that 
Specg(Hif) = U Spec^(7r«) 

A such that 

is a subset in M and has an upper bound. This follows from Lemma 8.5 (4) 
applied to each irreducible summand in (8.7.3). 

First, we consider the case where there exists a e M such that 

X{Z) = a mod Z for any A satisfying n„{X) ^ 0. (8.7.4) 

Then, the set ^ 

{A(Z) : AGt£,n,(A)^0} (8.7.5) 

is contained in Spec^(7ix) by (8.7.1), and is discrete by (8.7.4). Hence, it is 
a discrete subset of M with an upper bound. Thus, we can find hq € such 
that riTrilJ'o) 7^ and that Ho{Z) attains its maximum in (8.7.5). In turn, the 
evaluation map Swpp^{Ti.K) — > R, /i i-^ m(^) attains its maximum at /zo € 
Suppj(Wif) by (8.7.1) and Lemma 8.5 (3). Therefore, Hom(gp^^)(7r^^,Hii:) 7^ 
{0} by Lemma 8.6. Thus, we have shown m^g ^ 0, that is, tt'^^ occurs as a 
subrepresentation in H. 

Next, we consider the unitary representation tt' on 



the orthogonal complement of a subrepresentation tt^^ in Ti.. Then, the iiT-type 
formula (8.7.1) for {tt',H') holds if we set 



n^'(A) := 



n^(A)-l (A = /io), 
n^(A) (Xy^fio). 



Hence, by the downward induction on sup Spec^(Hif), we have n^(A) = mx 
for all A. ^ 

For the general case, let A be the set of complete representatives of {X{Z) G 
C mod Z : n.^{X) 0}. For each a & A, we define a subrepresentation Ha of 
Hhy 

Ha ■= 'mx'Kx (discrete Hilbert sum). 

A(Z)=amodZ 

Then, we have an isomorphism of unitary representations of G: 

H ~ ^2 ■ 

aeA 

Since Spcc^(7r^) C a + Z if and only if X{Z) = a mod Z by Lemma 8.5 (3), 
we get from (8.7.1) the following JsT-isomorphism 
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n^(A)4 (8.7.6) 

\{Z)=a mod Z 

for each a G A. Therefore, our proof for the first step assures n„{X) = m\ for 
any A such that A = a mod Z. Since a € A is arbitrary, we obtain Lemma in 
the general case. □ 

8.8 Proof of Theorem 8.3 

In this section, wc give a proof of Theorem 8.3. This is done by showing a more 
general formula in Lemma 8.8 without the scalar type assumption (8.3.1). 
Then, Theorem 8.3 follows readily from Lemma 8.8 because the assumption 
(8.3.1) makes dimTr^ — 1 and §(ai,...,aj)(M) ~ {m ~ '^j^j} i^'^^ (8.8.1) for 

notation). 

For a discussion below, it is convenient to use the concept of a multi- 
set. Intuitively, a multiset is a set counted with multiplicities; for example, 
{a, a, a, h, c, c}. More precisely, a multiset S consists of a set S and a function 
m : 5 ^ {0, 1, 2, . . . , oo}. If S' = {S, m'} is another multiset on S such that 
m'{x) < m{x) for all x £ S, we say §' is a submultiset of § and write §' C §. 

Suppose we are in the setting of Subsection 8.2 and recall t is an involu- 
tion of holomorphic type. For a Z\+ (6, t)-dominant weight n, we introduce a 
multiset S{fi) consisting of /i"'"(F, t'^)-dominant weights: 

S(/i) := IJ §(ai,...,oO(/^)' 

ai,...,aieN 

where we define the multiset S(ai,...,o;)(M) by 

{highest weight of irreducible F-modules occurring in 

7r'^„i TT* I {T counted with multiplicities}. (8.8.1) 

— 1^ j=i "'j'^j 

Because the central element Z = ^ Z of tc acts on the irreducible repre- 
sentation 77^ by the scalar n{Z) and because ^^{Z) = 1 for all j (1 < j < 0> 
any element v in S(ai,...,ao(A*) satisfies v{Z) = — o,j + ii{Z). Therefore, 
the multiplicity of each element of the multiset §(/u) is finite. 

Lemma 8.8. Let r be an involution of G of holomorphic type, and H = Gq. 
If TT^ is a (relative) holomorphic discrete series representation of G, then it 
decomposes discretely into irreducible H-modules as: 

m 

tt^Ih — ^ (discrete Hilbert sum). 
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Proof of Lemma 8.8. It follows from Fact 5.1 (1) that ttJ^ is (-ffnisr)-adniissible, 
and splits discretely into a Hilbert direct sum of irreducible unitary represen- 
tations of H. 

Applying Lemma 8.7 to H = Gq, we see that Lemma 8.8 is deduced from 
the following f^-isomorphism: 

TT^ ~ TT^ (algebraic direct sum). (8.8.2) 

The rest of the proof is devoted to showing (8.8.2). 

Since tt^ is a holomorphic discrete series, tt^ is isomorphic to the gener- 
alized Verma module N^{fi) = U{qc) ®u{tc+p+) ^ ^ g-module, which in 
turn is isomorphic to the f-module /^(p-) (Xi tt' . 

According to the decomposition p_ = p!l © pl"^ as F-modules, we have 
then the following {^-isomorphism: 

ttB 5(p_) ® TT* ^ 5(p:) ® Sipzn ® ttI . (8.8.3) 

Now, we consider the Hermitian symmetric space ff^^/G"^'*, for which the 
complex structure is given by the decomposition gj^ = 6^ ® p~^ ® p~^ (see 
(8.2.2)). Then, the Hua-Kostant Schmid formula ([78, Behauptung c]) ap- 
plied to C^^/C^'^ decomposes the symmetric algebra -S'(pl'^) into irreducible 
t^-modules: 

SiPZn^ (8.8.4) 

ai>--->ai>0 
ai,...,OieN 

It follows from the definition of §(/i) that we have the following irreducible 
decomposition as F-modules: 

Combining this with (8.8.3), we get a 6'^-isomorphism 

S{pl)®7r'J. 

Next, we consider the Verma module N^"^ (u) = U{g'Q ^uit^+v:^) of the 
subalgebra . Then, tt^^ is the unique irreducible quotient of N^^ {v). We 
shall show later that N^'' {v) is irreducible as a g'^-module, but at this stage 
we denote by irf^ , tt^, , tt^,, , . . . the totality of irreducible subquotient modules 
of [v). (There are at most finitely many subquotients, and all of them are 
highest weight modules.) Then, as F-modules, we have the following isomor- 
phisms: 

— 'K-iy TT^/ T^yii • • • . 
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Therefore, we get a F-isomorphism: 

<^ «©7r«:©7r«: ©•••). 

Accordingly, the restriction 7rj^|i/ spUts discretely into irreducible unitary rep- 
resentations of H by Lemma 8.7: 

Since n'j^ is a (relative) holomorphic discrete series representation of G, all 
irreducible summands in the right-hand side must be (relative) holomorphic 
discrete scries representations of H by Fact 5.1 (1). Therefore, (u) is 
irreducible, and the other subquotients nf,, , nf,,, , . . . do not appear. Hence, 
the f^-structures of the both sides of (8.8.2) are the same. Thus, Lemma 8.8 
is proved. □ 

8.9 Proof of Theorem 8.4 

For two irreducible representations tt^^ and tt^^, we define a multiset S(/Lti, /X2) 
consisting of Z\^(6, t)-dominant weights by 

§(Mi,M2):= U §(ai,...,ar)(/"l,M2), 

ai>'">af>0 
ai,...,ai-eN 

where §(ai....,a,-) (a*1) M2) is the multiset consisting of highest weights of irre- 
ducible 6-modules occurring in tt* ^ r _ ® 7rf, ® 7rf, counted with multi- 

plicities. 

Theorem 8.4 is derived from the following more general formula: 

Lemma 8.9. The tensor product of two (relative) holomorphic discrete series 
representations and ir^^ decomposes as follows: 

i/6S(Mi,;i2) 

Proof. We define two injcctivc maps by: 

diag : p+ ^ p+ ® p+ , X^{X,X), 
diag' : p+ p+ e p+ , X^{X, -X) . 

It then follows that we have {-isomorphisms: 
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~5(diag(p_))0 5(diag'(p_)) 
~ 5(diag(p_))®7r^ 



ai>-">a^>0 
ai,...,ai-£N 



This brings us the following 6-isomorphisms: 




^2 — 



5(diag(p_))®7r^ 







i'eS(/ii,/:i2) 



The rest of the proof goes similarly to that of Lemma 8.8. □ 

8.10 Restriction U{p, q) I U{p - 1, q) and SO{n, 2) i SO{n - 1, 2) 

Suppose (G, H) is a reductive symmetric pair whose complexification {gc, f)c) 
is one of the following types: 

(s[(n, C), fl[(n - 1, C)) (or {gl{n, C),gl{l, C) + fl[(n - 1, C))), 

(so(n,C),so(n- 1,C)). 

As is classically known (see [83]), for compact {G,H) such as (C/(n),[/(l) x 
U{n — 1)) or {SO{n), SO{n — 1)), any irreducible finite dimensional repre- 
sentation TT of G is multiplicity- free when restricted to H. For non-compact 
{G,H) such as {U{p,q),U{l) x U{p - l,q)) or {SO{n,2), SO{n - 1,2)), an 
analogous theorem still holds for highest weight representations tt: 

Theorem 8.10. // (fl.t)) = (u(p, g), u(l)-hu(p-l, g)) or (so(n, 2), so(n-l, 2)), 
t/;,eT?, any irreducible unitary highest weight representation of G decomposes 
discretely into a multiplicity-free sum of irreducible unitary highest weight 
representations of H. 

In contrast to Theorem A, the distinguishing feature of Theorem 8.10 is 
that TT is not necessarily of scalar type but an arbitrary unitary highest weight 
module. The price to pay is that the pair (G, H) is very special. We do not give 
the proof here that uses the vector bundle version of Theorem 2.2 (see [49]). 
Instead, we give an explicit decomposition formula for holomorphic discrete 
series tt. The proof of Theorem 8.10 for the case (G, H) = {SOo{n, 2), SOo{n— 
1,2)) can be also found in Jakobsen and Vergne [31, Corollary 3.1]. 

8.11 Branching law for U{p,q) J, U{p — l,q) 



This subsection gives an explicit branching law of a holomorphic discrete scries 
representation oi G = U {p, q) when restricted to H = U{1) x U(p— l,q). 
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Owing to (8.3.2), such ir^ is parametrized by /i = (/^i, . . . , /Xp+q) e Z^'+' 
satisfying 

Ml > • • • > Mp, Mp+l > • • • > fJ-p+q, fJ-p+g > Ml + P + 9 • 

Here is the formula: 

Theorem 8.11 (Branching law U{p,q) i U{p — l,q))- Retain the above 
setting. Then, the branching law of of the restriction to the subgroup H is 
multiplicity-free for any fi; it is given as follows: 

oo 

^ V^® IT f^U{p-l,q) 

^'^c\H- 2^ 2^ (LeLi f^i-^U A.-a^^(A„...,A„A,+i,...,Ap+,) 

a=0 Mi>A2>At2>--->Ap>/ip 

Ap+i > • • •> Ap-(-, >/ip-|-<, 

S?=i(Ap+i-Aip+i)=o 

(8.11.1) 

Proof For (G, if) = {G, C) = {U{p, q), U{1) x U{p - 1, q)), we have 

G"' ~f/(l,g)xf/(p-l), 
Hr\K {=K'' = K''^) ~ [/(I) X U{p - 1) X [/(g), 

r = t, and 

A+{pl\ e) = {ei - Cp+i ■.l<i<q} 

by using the standard basis of Z\(g, t) = {±(ei — e^) : 1 < z < j < p + q}. 
Thus, I = R-rankG"^^ = 1 and vx = ei — Cp+i. Hence, the 6'^-type formula 
(8.8.4) amounts to 



%=^)-0'r?r(f.-ep,O 

oo 

^0C_„KlK7rJ^„;^; .^0) (8.11.2) 

0=0 

as iJ n iiT ~ [/(I) X U{p — 1) X U{q) modules. Here, 1 denotes the trivial one 
dimensional representation of U{p— 1). 

On the other hand, we recall a classical branching formula U{p) J, U {p— 1): 

([il>A2>(li2>--->Ap>(Llp 

whereas the classical Fieri rule says 

u{q) ^vii) _ n\ u(q) 



(Ap+i,---,Ap+<,) ■ 

Ap+i>Mp+i>'"> Ap+,>/ip_|_, 
Ei=i(Ap+i-Mp+i)=a 
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These two formulae together with (8.11.2) yield the following F-isomorphisms: 

0((C_„ ^ 1) ® <,?,!..„^)k(i)x^(p-i)) ^ (<2....o) ® Ct,...,...,)) 



a=0 



U{p-1) f^^Uig) 

Xp+i,...,Ap+<,; 
a=0 /^i>A2>(U2>"->Ap>/Xp 



In view of the fi'^-isomorphisms 

and 7V3^(!^) ~ S{p'L) (8) tt'"^, we have now shown that the F-structure of tt^ 
coincides with that of 

oo P P 

N^\Y.t,i-Y,Xi-a,X2,...,Xp+,). 

a=0 Ati>A2>Ai2>'">Ap>/jp i=l i=2 

Ei=l(^p+i-/'p+«)=« 

As in the last part of the proof of Theorem 8.3, we see that any generalized 
Verma module occurring in the right-hand side is irreducible (and is isomor- 
phic to the underlying {q^, _ffn if) -module of a holomorphic discrete series of 
H). Therefore, Theorem follows from Lemma 8.7. □ 



9 Appendix: Associated Bundles on Hermitian 
Symmetric Spaces 

In this Appendix, we explain standard operations on homogeneous vector 
bundles. The results are well-known and elementary, but we recall them briefly 
for the convenience of the reader. The main goal is Lemma 9.4 which is used 
to verify the condition (2.2.2) in Theorem 2.2. 

9.1 Homogeneous vector bundles 

Let M be a real manifold, and V a (finite dimensional) vector space over 
C. Suppose that we are given an open covering {Ua} of M and transition 
functions 

satisfying the following compatibility conditions: 

9ai3 901 Qja = id on UaOUisnUj ; g^a = id on Ua ■ 
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A complex vector bundle V over M with typical fiber V is constructed as the 

equivalence class of IJ„(?7a x V), where (x, v) G Up x V and {y, w) E Ua x V 
are defined to be equivalent ii y = x and w = gaf3{x)v. Then, the space of 
sections r{M, V) is identified with the collection 

{(/a) : fa G C°^{Ua,V), U{x) = ga/}{x)U{x), for xeU^nUp}. (9.1.1) 

If M is a complex manifold and if every is holomorphic (or anti- 
holomorphic) , then V — > M is a holomorphic (or anti-holomorphic, respec- 
tively) vector bundle. 

Next, let G be a Lie group, K a closed subgroup of G, and M := G/K 
the homogeneous manifold. Then, we can take an open covering {Ua} of M 
such that for each a there is a local section Lp^ : Ua ^ G of the principal 
bundle G — * G/K. Given a representation x ■ K ^ GLc{V), we define the 
homogeneous vector bundle, V := G (X) Then V is associated with the 
transition functions: 

5a/3 -.UanUfS ^ GLc{V), ga0{x) := X{'fiaix)~'^(p0{x)) . 

The section space r{M, V) is identified with the following subspace of C°°(G, V): 
{/ G G~(G, V) : f{gk) = x-\k)f{g), for 5 G G, e if } . (9.1.2) 

9.2 Pull-back of vector bundles 

Let G' be a Lie group, K' a closed subgroup of G', and M' := G'/K' the ho- 
mogeneous manifold. Suppose that ct : G' ^ G is a Lie group homomorphism 
such that cr{K') c K. We use the same letter a to denote by the induced map 
M' M, g'K' ^ (j{g')K. Then the pull-back of the vector bundle V ^ M, 
denoted by cr*V — > M', is associated to the representation 

Xoa:K'^GLciV). 

Then we have a commuting diagram of the pull-back of sections (see (9.1.2)): 

a*: r{M,V) ^ r{M',a*V), (/«)« (/« o a)„ , 

n n 

a* :C°°{G,V) ^ C°°{G\V), f ^ foa. 

9.3 Push-forwcird of vector bundles 

Suppose that V and W are complex vector spaces and that ^ : V ^ W 
is an anti-linear bijective map. Then, we have an anti-holomorphic group 
isomorphism 



GLc{V) GLciW) , g^g^:=iogoC' 
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Let V — > M be a complex vector bundle with transition functions ga(3 ■ Ua n 
Up ^ GLc{V). Then, one constructs a complex vector bundle — > M 
with the transition functions : Ua r\ U/j GLc{W). We have a natural 
homomorphism 

:r(M,V)->r(M,$,V), (/a)^($o/„), 

which is well-defined because the compatibility condition in (9.1.1) is satisfied 

as follows: If x G UaOUp then 

9i0{x){i° f0){x) = {^o goci3{x)o^-^)(^o f0){x) = ^ogafi{x)fi3{x) =^o/„(x). 

If V is the homogeneous vector bundle G y^K (X; ^) associated to a rep- 
resentation X : if — !■ GLc{V), then is isomorphic to the homogeneous 
vector bundle G Xk {x.^, W) associated to the representation 

X^:K^ GLc{W) , k ^ xHk) := ^ o x{k) o ■ 

9.4 A sufficient condition for the isomorphism ^»cr*V ~ V 

We are particularly interested in the case where G' = G, K' = K,V = W = C 
and ^(z) := z (the complex conjugate of z) in the setting of Subsections 9.2 
and 9.3. 

By the identification of GLc(C) with C^, we have = g for g G 
GLc{V) ~ C^. Then, coincides with the conjugate representation 

X-.K^ GLciW) ~ C^ 

for X € ilom{K,C^). Thus, we have an isomorphism of G-equivariant holo- 
morphic line bundles: 

C*o-*V~ G Xa: (xW,C) (9.4.1) 

with the following correspondence of sections: 

o a* : r(M, V) ^ r(M, ^,a*V) , (/„) ^ {J^) . 

We now apply the formula (9.4.1) to the setting where M = G/K is an 
irreducible Hermitian symmetric space. 

Lemma 9.4. Let x '■ K ^ be a unitary character. We denote by V the 
homogeneous line bundle G Xk (X;*C). Suppose a is an involutive automor- 
phism of G of anti-holom,orphic type (see Definition 1.4)- Then we have an 
isomorphism of G-equivariant holomorphic line bundles: ^*cr*V V. 
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Proof. In view of (9.4.1), it suffices to show xo a = x. As the character % of 

K is unitary, wc have x(fc) = x{^-^^) for ^-ny k E K. Let Z be a generator of 
the center c{t) of t. Since a is of anti-holomorphic type, we have aZ = —Z, 
and then 



Xoa(expiZ) =x(expHZ)) = x(exptZ) (t € M) . 

On the other hand, if A; € [K, K], then x ° f (fc) = 1 = x(^) because [K, K] is 
a connected semisimple Lie group. As K = expc(6) • [K,K], we have shown 
X o cr = X- Hence Lemma. □ 
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